Most active commenters
  • apeace(3)

←back to thread

1624 points yaythefuture | 46 comments | | HN request time: 1.124s | source | bottom

Saw https://news.ycombinator.com/item?id=32261868 from a couple weeks ago and figured I'd share my own story.

3 weeks ago, I woke up to a pissed off customer telling me her payments were broken. My startup uses Stripe Connect to accept payments on behalf of our clients, and when I looked into it, I found that Stripe had decided to deactivate her account. Reason listed: 'Other'.

Great.

I contact Stripe via chat, and I learn nothing. Frontline support says "we'll look into it." Days go by, still nothing. Meanwhile, this customer is losing a massive amount of business and suffering.

After a few days, my team and I go at them from as many angles as possible. We're on the phone, we're on Twitter, we're reaching out to connections who work there / used to work there, and of course, we reach out to patio11. All of these support channels give us nothing except "we've got a team looking into it". But Stripe's frontline seems to be prohibited from offering any other info, I assume for liability reasons. "We wouldn't want to accidentally tell you the reason this happened, and have it be a bad one."

We ask: 1. Why was this account flagged? "I don't have that information" 2. What can we do to get this fixed? "I don't have access to that information. 3. Who does? "I don't have access to that information" 4. What can you do about this? "I've escalated your case. It's being reviewed."

I should mention at this point that I've been running this business since 2016, my customers have been more or less the same since then, and I've had (back when it was apparently possible) several phone conversations with Stripe staff about my business model. They know exactly who our customers are and what services we offer, and have approved it as such.

After a week of templated email responses and endless anxiety, we finally got an email from Stripe letting us know that they had reviewed the account and reactivated it. We never got a reason for why any of this had happened, despite asking for one multiple times. Oh well, still good news right? Except nope, this was only the beginning.

This morning I woke up to an email that about 35% of my client accounts had been deactivated and were "Under review", the kicker here being that one of those accounts is the same one they already reviewed last week! This is either the work of incompetent staff or (more likely) a bad algorithm. No reasonable human could make this mistake after last week's drama.

So currently, my product doesn't work for 35% of my customers. Cue torrent of pissed off customer emails.

And the best part is, this time I have an email from Stripe this time: Apparently these accounts are being flagged, despite the notes on our file, and despite the review completed literally last week, as not in compliance with Stripe's ToS. They suggest that if I believe this was done in error, I should reach out to customer support. Oh, you mean the same customer support that can't give me literally any information at all other than "We have a team looking into it"? The same customer support that won't give me any estimates as to how long it's going to take to put this fire out? The same customer support that literally looked into this a week ago and found no issues!?

I feel like I'm going crazy over here. These accounts have hundreds of thousands of dollars in them being held hostage by an utterly incompetent team / algorithm that seems to lack any and all empathy for the havoc they wreak on businesses when they pull the rug out from under them with no warning, nor for the impact they have on customers when they all of a sudden lose all ability to make money. And all that for an account that has been using Stripe for nearly 7 years without issue!

This goes so far beyond "customer support declining at scale." If lack of customer support means that critical integrations start to fail, that's not a customer support failure, that's a fundamental business failure.

1. apeace ◴[] No.32855524[source]
I had a very similar experience, except it was on our launch day.

We hadn't charged a single live customer yet, but we had done plenty of tests using the Stripe testing environment. So we go live with a huge launch event, and we have customers signing up in droves. When they get to the last step -- payment -- they get an error.

Logging in to the dashboard I didn't see any indication that there was anything wrong with our account. No alerts or notices. We had already gone through the approval process you go through when signing up, and been told we were approved.

The thing that surprised me the most was that there was just no indication anywhere that our account would not be able to charge cards. Wouldn't it make sense for there to be an indicator somewhere that just says "Not ready yet"?

Apparently, they had never even begun reviewing/vetting us since the time we signed up for the account months earlier. We reached out to customer support and it took them about two weeks to get us activated. And, similar to OP, they never gave us a shred of information about what was going on. I still don't know to this day what the issue was.

Next time I build something with Stripe I'm going to test it in production before launching, with my own real credit card!

replies(5): >>32855809 #>>32856401 #>>32857067 #>>32858268 #>>32863136 #
2. mgkimsal ◴[] No.32855809[source]
> Next time I build something with Stripe I'm going to test it in production before launching

I'm genuinely curious why you wouldn't have done that anyway? I pretty much always do, precisely so I can experience a full end to end user experience.

replies(5): >>32856045 #>>32856085 #>>32856098 #>>32856168 #>>32863119 #
3. pc86 ◴[] No.32856045[source]
I mean yes, shame on them for not charging themselves $1 or something in production, but still, it's insane for Stripe to have no indication on the dashboard that you're not actually able to charge real cards real money.
4. londons_explore ◴[] No.32856085[source]
Buying your own products inflates revenue figures (and is therefore pretty much prohibited in listed companies). Many investors don't like it either.

Sure, you can do it, but you'll have to forever have a note in your accounts package saying 'this revenue isn't actual revenue'.

It really upsets those who want the accounts to match up to the cent at the end of the year...

replies(2): >>32857598 #>>32858335 #
5. mtmail ◴[] No.32856098[source]
Stripe advises against that (I'm not saying it makes sense)

"Do not use real card details. Testing in live mode using real payment method details is prohibited by the Stripe Services Agreement. Use your test API keys and the card numbers below."

https://stripe.com/docs/testing

replies(6): >>32857768 #>>32858479 #>>32859573 #>>32861600 #>>32863865 #>>32877263 #
6. apeace ◴[] No.32856168[source]
The funny part is, I thought that if I did a bunch of real transactions and then reversed them all, that our account might get flagged!

That's in addition to explaining to our investors that we ended our launch day at -$1,000 due to all the chargeback fees because some dumb developer doesn't know what a testing environment is. And in addition to the fact that Stripe recommends against you testing in that way.

I did get a full end-to-end experience, in my production environment, with literally one variable changed: using the testing Stripe API key instead of the production one.

I don't know, it didn't seem like a crazy way of testing at the time. When you consider that the testing environment worked perfectly, and there was no indication, whatsoever, anywhere that we would not be able to charge cards, it kind of felt like how you're supposed to do it.

But clearly I was wrong :) The way you find out if your account has any issues is by charging a real card, and if it works, reversing it, and if it doesn't work, waiting a couple weeks with no information on what's going on. Lesson learned!

replies(3): >>32857353 #>>32859084 #>>32863439 #
7. tremon ◴[] No.32856401[source]
Next time I build something with Stripe

I'm curious why you would still consider them a trustworthy business partner after what happened?

replies(3): >>32856824 #>>32860318 #>>32860936 #
8. cpurdy ◴[] No.32856824[source]
Exactly. After reading this thread, the chance that I would ever choose to use Stripe is exactly 0%.

What a disaster.

9. theflyingkiwi42 ◴[] No.32857067[source]
That's how AMEX shut us our corporate credit card account overnight. We launched a new shopping cart and tested it live in production with our own AMEX card. Mind you that we used that card for over $250k a month (never a late payment) and this was a $99 charge that we simply forgot to refund.

No appeal process, nothing. Caused significant short term cash flow issues. So be very careful using your own card.

replies(1): >>32858029 #
10. krisoft ◴[] No.32857353{3}[source]
> charging a real card, and if it works, reversing it

Why would you reverse it? If you can’t afford to consume one unit of whatever you are selling are you really in a good place to be in that business?

I guess there can be rare exceptions where the business sell only a handfull of high ticket items. But then again probably Fincantieri does not let you put your bespoke mega yacht on a card through a web transaction.

replies(3): >>32857426 #>>32860770 #>>32861097 #
11. apeace ◴[] No.32857426{4}[source]
We are an internet service provider and we charge a monthly fee for our service. So the real thing I would've been doing was reversing the first month's charge and then cancelling the recurring subscription. Which is exactly what I'd recommend you do if you are launching a new subscription service on Stripe!
replies(3): >>32858947 #>>32859749 #>>32860273 #
12. krisoft ◴[] No.32857598{3}[source]
I smell a made up problem. Do you really think the CEO of Chipotle can’t buy a burrito from one of their branches without accountants getting a heart attack?

They just walk in and buy one. They in their personal capacity end up richer by a burrito and some invaluable experience. The company ends up richer by the price of a burrito. If this kind of “revenue inflation” matters to anyone then either the CEO has a bad burrito addiction or the company wasn’t transacting enough anyway or both.

replies(1): >>32857958 #
13. remus ◴[] No.32857768{3}[source]
Frankly that's a ridiculous clause. If you make a change (like switching from testing to live in the stripe dashboard) it needs to be tested, as OP's comment nicely demonstrates.
14. londons_explore ◴[] No.32857958{4}[source]
Where I worked, the CEO would walk in and get a free burrito. It's another kind of headache (technically that burrito is pay, and needs to be taxed and declared as CEO pay), but that was considered preferable to revenue inflation which is fraud rather than dodging taxes.
replies(2): >>32858876 #>>32861638 #
15. ValentineC ◴[] No.32858029[source]
AMEX used to have major issues with consumers trying to "pay" themselves — the Singapore version put me in financial review for a PayPal transaction that I had accidentally paid to the wrong address, and was subsequently cancelled and refunded — which to AMEX looked like I was trying to "pay" myself.

I don't know if that's still the case, but these days I'm wary about using AMEX on any sort of money transfer service.

16. demarq ◴[] No.32858268[source]
I'm surprised there is a next time.

I think that's the biggest thing I see on this thread. That Stripe gets the chance of a next time. This validates what Stripe is doing to developers.

replies(1): >>32859664 #
17. imperfect_blue ◴[] No.32858335{3}[source]
Debit revenue, credit employee expense claims, journal entry: adjustment for test purchases made. Pretty standard, doesn't affect your numbers at all, no need to make any note in your statement (which you wouldn't have to do anyway because it'll never meet the threshold for materiality). Its not some sort of difficult accounting problem.
replies(1): >>32858461 #
18. ansible ◴[] No.32858461{4}[source]
Yes, exactly. I would think that reversing charges raises a flag with whatever payment processor or bank that sees it.
replies(1): >>32858543 #
19. yamtaddle ◴[] No.32858479{3}[source]
It's bullshit. Anyone with experience testing things in this area is 100% gonna try a real card at least after any major changes (and certainly before a launch of a new product). You aren't actually sure it's right (and so, with something so entirely connected to revenue, can't go the fuck to sleep) until a real charge has gone through.

They probably mainly just don't want people running shitloads of automated-test charges and issuing refunds for all of them. It may technically be against the TOS but there's no way they actually care if you run the occasional real-credentials test, especially if you don't refund it.

replies(2): >>32860202 #>>32860363 #
20. imperfect_blue ◴[] No.32858543{5}[source]
There's no need to reverse it even, just go through your standard expense claims procedure or product refund procedure.
21. dylan604 ◴[] No.32858876{5}[source]
This is were some fucking common sense needs to break out. If the CEO at a food chain cannot legitimately eat there like a regular customer, then we've got some jacked up bullshit regulations. If the CEO is driving to each branch and order 1000 items, then sure, that again should be common sense of something.

More and more, we just keep acting dumber as "tech" is more and more engrained

22. sgc ◴[] No.32858947{5}[source]
Just make a hidden $1 subscription and cancel it without reversing it, then deactivate the item.
replies(1): >>32860289 #
23. bcrosby95 ◴[] No.32859084{3}[source]
Yes. When interfacing with a complex system, especially one you don't control, always do at least one live test.
24. parkerhiggins ◴[] No.32859573{3}[source]
This cause exists not for product testing and QA but rather to prevent payment details for being abused in the name of “testing”.

Everyone runs 1-3 “real transactions” as a “real customer” when getting ready for a launch with QA.

This cause exists for Stripe to point to for excuses people make when attempting to wash transaction or test stolen CC info in a prod environment (which doesn’t work with a test api key)

25. disqard ◴[] No.32859664[source]
"What is water?"
replies(1): >>32861195 #
26. Twirrim ◴[] No.32859749{5}[source]
Why not just claim the expense on your card as business expense, and pay yourself back?
27. mgbmtl ◴[] No.32860202{4}[source]
Refunds used to be free, but for a few years now Stripe keeps the transaction fees.

Although I guess doing lots of refunds would mess with anti-fraud and other systems.

I just do special pricing/discounts to do a $5 transaction (too small could look like credit card fraud).

28. w-ll ◴[] No.32860273{5}[source]
Another good reason to do a Live charge even for subscriptions (add a coupon code to make it like $1 if you really need to) but is to test the credit card expiring after the fact. I've done a lot of billing code an none of the sandboxes really let you test a card that works for N months, and then expires.
29. bigiain ◴[] No.32860289{6}[source]
I still got caught when I did that once. $1 worked just fine. The first real customer $249 charge failed. :-(

Test in production. Do real dollar value tests. If you can test with different cards with different security levels, try a Visa 3D Secure and a 2FA Amex charge. Personally I do them and then get reimbursed (or do them directly on a company credit card) rather than start out a production payment history with refunds, not sure if that matters but I figure if it does it's got to be a bad signal so I may as well avoid it.

30. lmm ◴[] No.32860318[source]
They might still be the best of a bad bunch. Payment processing is a de facto cartel (probably driven by it being an actual cartel at the Visa/Mastercard level).
31. bigiain ◴[] No.32860363{4}[source]
> especially if you don't refund it.

That's the trick. Let Stripe put the money into your bank account and let them think its a real sale (not a production test), then refund the money out of the bank account back to the person who's card was used instead of refunding or reversing the charge. Stripe don't need to know about that.

(You also owe yourself "due diligence" in knowing that you _can_ refund or reverse charges, so you'll also do that test in production as well, but don't make it one of the first transactions you ever do in prod. I try to stretch that test out until after a 60 day window past the first few dozen real sales have gone through, under the assumption that by then Stripe (or whoever) will have seen a bunch of payments go through and not be challenged when the CC statements arrive, and that'll have sent some "probably not a new fraudulent merchant" signals into their systems.)

replies(1): >>32860999 #
32. Aeolun ◴[] No.32860770{4}[source]
> Why would you reverse it?

Why would you not? Stripe still keeps the transaction cost, so clearly they anticipate this happening.

33. matiasfernandez ◴[] No.32860936[source]
We've been moving away from Stripe and Stripe-backed (e.g. Shopify) payment gateways pretty much everywhere we can for this same reason. We only have a few integrations left connected to Stripe, specifically one with Tripleseat, which we use for our event management and have not migrated yet. Choosing Stripe has been the most costly decision we've ever made and we're not a small company nor a startup.
replies(1): >>32861749 #
34. majormajor ◴[] No.32860999{5}[source]
This is what my team always did for QA when I worked in a space using payment processors.

No need to roll back the transactions directly and risk flagging anything.

Plus, it's often useful having a real, live, paid and in good standing account and/or purchase in the system for further testing steps! (More true in the subscription space than the one-time purchase space... but even there... it's probably worth testing your refund flow end to end periodically eventually, once you have a healthy set of traffic under your belt to avoid standing out.)

35. Psychlist ◴[] No.32861097{4}[source]
And this is exactly what I told a friend of my to do when he installed a public car charger. Before he told anyone it was live, charge his car using his normal customer account. Turned out he couldn't create one because his phone was already in as a supplier.

After some eye-rolling he agreed to buy a burner phone, add a prepay SIM, and whaddayano the bloody charger did not work. Two months later it works, for most people, who have recent cars that are not Teslas. Sigh. "welcome to our product, you're a beta tester. Maybe an alpha tester. We hope it works!".

replies(1): >>32897190 #
36. tobinfekkes ◴[] No.32861195{3}[source]
I just played that video at an outdoor movie night. It gives me the feels every time. Great video!
37. paul_f ◴[] No.32861600{3}[source]
Run a real card for a real amount and write a check to the person whose card you used. Do this multiple times with different types of cards.
38. jhugo ◴[] No.32861638{5}[source]
The CEO buying a burrito and paying for it with their own money because they want to eat a burrito is not revenue inflation. I can't even imagine what kind of fantasy world you would have to live in to believe this rubbish.
39. jkira ◴[] No.32861749{3}[source]
What payment processor did you switch to?
replies(1): >>32885420 #
40. GrumpyNl ◴[] No.32863119[source]
Always test your life platform, doenst matter if its payments or software, always test life, i can not emphasize that enough.
41. tuukkah ◴[] No.32863136[source]
Same happened to me a year ago. My customer was on a tight schedule to run a trial, and postponing the launch on the launch day because of Stripe ruined their trial.
42. vishnugupta ◴[] No.32863439{3}[source]
> reversing it

This isn't recommended. Also, presumably by end-to-end you meant post auth business flow too. Such as reconciliation, settlement, generating tax receipts, invoices etc.,

In general, all the data in production should be real, as much as possible.

If the amount is big (e.g., say jewellery story) then figure out a way to recoup/refund the money out of band in a way that leaves no trace in production system. E.g., reimburse it, put it under QA budget or some such.

43. Thorrez ◴[] No.32863865{3}[source]
The testing doc says it's against the agreement, but from a 1 minute look at the agreement, I can't find where the agreement bans testing in live mode. Do you see it?
44. mgkimsal ◴[] No.32877263{3}[source]
Others replied already, but I'll chime in too.

There's "testing"

* manual during development

* automated testing

And there's "testing"

* smoke checking processes in production

In most cases, the "test in production" is more of a "validation" that end to end experience works. And there probably shouldn't really be much of a way to distinguish between a "test" in production and a "validation" of the purchase process.

45. matiasfernandez ◴[] No.32885420{4}[source]
Our main bulk of transactions get processed with 4 different local payment providers (Accepta, Bank Tech, Dynamics Payments, and Evertec), one of which integrates with Authorize.net, which in turn integrates with Shopify as a custom gateway. For all of them, we were able to negotiate really good fees that are half of what we currently pay with Stripe.

My recommendation is to look into your local providers and see if any can integrate with payment APIs like Authorize.net. These providers really want your business and care enough to at least not let you go bankrupt. The bar is low!

46. e-clinton ◴[] No.32897190{5}[source]
There are countless of virtual phone apps which allows calling and SMS for a few bucks a month. That’s what I’m doing to create “test” accounts on our stripe prod environment for something I’m building.