Most active commenters
  • brightball(5)
  • canadiantim(4)

←back to thread

1624 points yaythefuture | 40 comments | | HN request time: 2.677s | source | bottom

Saw https://news.ycombinator.com/item?id=32261868 from a couple weeks ago and figured I'd share my own story.

3 weeks ago, I woke up to a pissed off customer telling me her payments were broken. My startup uses Stripe Connect to accept payments on behalf of our clients, and when I looked into it, I found that Stripe had decided to deactivate her account. Reason listed: 'Other'.

Great.

I contact Stripe via chat, and I learn nothing. Frontline support says "we'll look into it." Days go by, still nothing. Meanwhile, this customer is losing a massive amount of business and suffering.

After a few days, my team and I go at them from as many angles as possible. We're on the phone, we're on Twitter, we're reaching out to connections who work there / used to work there, and of course, we reach out to patio11. All of these support channels give us nothing except "we've got a team looking into it". But Stripe's frontline seems to be prohibited from offering any other info, I assume for liability reasons. "We wouldn't want to accidentally tell you the reason this happened, and have it be a bad one."

We ask: 1. Why was this account flagged? "I don't have that information" 2. What can we do to get this fixed? "I don't have access to that information. 3. Who does? "I don't have access to that information" 4. What can you do about this? "I've escalated your case. It's being reviewed."

I should mention at this point that I've been running this business since 2016, my customers have been more or less the same since then, and I've had (back when it was apparently possible) several phone conversations with Stripe staff about my business model. They know exactly who our customers are and what services we offer, and have approved it as such.

After a week of templated email responses and endless anxiety, we finally got an email from Stripe letting us know that they had reviewed the account and reactivated it. We never got a reason for why any of this had happened, despite asking for one multiple times. Oh well, still good news right? Except nope, this was only the beginning.

This morning I woke up to an email that about 35% of my client accounts had been deactivated and were "Under review", the kicker here being that one of those accounts is the same one they already reviewed last week! This is either the work of incompetent staff or (more likely) a bad algorithm. No reasonable human could make this mistake after last week's drama.

So currently, my product doesn't work for 35% of my customers. Cue torrent of pissed off customer emails.

And the best part is, this time I have an email from Stripe this time: Apparently these accounts are being flagged, despite the notes on our file, and despite the review completed literally last week, as not in compliance with Stripe's ToS. They suggest that if I believe this was done in error, I should reach out to customer support. Oh, you mean the same customer support that can't give me literally any information at all other than "We have a team looking into it"? The same customer support that won't give me any estimates as to how long it's going to take to put this fire out? The same customer support that literally looked into this a week ago and found no issues!?

I feel like I'm going crazy over here. These accounts have hundreds of thousands of dollars in them being held hostage by an utterly incompetent team / algorithm that seems to lack any and all empathy for the havoc they wreak on businesses when they pull the rug out from under them with no warning, nor for the impact they have on customers when they all of a sudden lose all ability to make money. And all that for an account that has been using Stripe for nearly 7 years without issue!

This goes so far beyond "customer support declining at scale." If lack of customer support means that critical integrations start to fail, that's not a customer support failure, that's a fundamental business failure.

1. brightball ◴[] No.32855155[source]
Stuff like this is why I think a business should have a system that abstracts away the payment processor.

I use Stripe for invoices, but I can easily send an invoice through another platform if needed.

For processing transactions on the web, I would always lean toward using a service like ChargeBee that allows me to setup multiple payment gateways.

Getting off the ground quickly is one thing, but the moment that you have reliable revenue is the moment that you need to put some serious emphasis on redundancy across your business to plan for disasters, outages, etc. It's worth it to pay the fees to maintain a 2nd (or 3rd) payment processor once you have that type of revenue coming in.

replies(8): >>32855295 #>>32855442 #>>32855463 #>>32855474 #>>32855479 #>>32855597 #>>32855655 #>>32855991 #
2. BiteCode_dev ◴[] No.32855295[source]
And if ChargeBee bans you, you are still dead because that's a single point of failure. No, you should have several payment processors, and rotate them regularly to check they work.
replies(4): >>32855332 #>>32855344 #>>32855987 #>>32865985 #
3. eropple ◴[] No.32855332[source]
It's been a while since I dealt with this side of ecommerce - how can you have several payment processors, and rotate them, without having to handle card data yourself? (Most folks aren't really equipped to do that.)
replies(3): >>32855404 #>>32855523 #>>32856036 #
4. brightball ◴[] No.32855344[source]
Under ChargeBee's terms, they must provide you with your data for up to 120 days following any termination. So there's that at least.

> 7.2.3. Data Export Following termination or expiration of a Subscription, We will retain that Account’s Service Data for one hundred twenty (120) days from such date of termination or expiration (“Data Retention Period”).

I mean, ideally we need an open source PCI compliant equivalent of ChargeBee so that you can 100% own your customers payment information.

That's the way this problem really gets solved, but the security surface for that open source project is going to be a challenge.

replies(2): >>32855625 #>>32859526 #
5. ◴[] No.32855404{3}[source]
6. yaythefuture ◴[] No.32855442[source]
We're built on Rails which has the extreme luxury of being able to use ActiveMerchant, which does exactly this. The problem is, abstraction falls apart when you're using functions that are specific to a product. Stripe Connect is is nearly impossible to replicate with an adapter.
7. symfrog ◴[] No.32855463[source]
This is one of the reasons that I usually use https://killbill.io/
replies(2): >>32855589 #>>32855636 #
8. mattbee ◴[] No.32855474[source]
Yup I did this at my last business in 2008; instead of changing payment processors we abstracted them away behind a pretty small API that also stored and charged card numbers. We could flip back and forth between two. PCI DSS gradually became a bummer around that time but it was a very slow introduction which we were able to cope with.
9. jstummbillig ◴[] No.32855479[source]
> Stuff like this is why I think a business should have a system that abstracts away the payment processor.

Realistically, more humanly, payment processors and other big tech companies that are basically societies digital gum and infrastructure can simply not be tasked with making these calls. I also don't think they are very keen to do it but in the absence of timely regulation they must.

There have to be more rigorous ground rules (what is the business allowed to do, what must they do, what is the user allowed to do, and what are they entitled to), by law, and quickly.

replies(2): >>32855804 #>>32856053 #
10. BiteCode_dev ◴[] No.32855523{3}[source]
You use several API, attach one provider per customer to divide the risks. When one provider bails out, you onboard new customers and new payment on the other payment provider.

Then you only loose the recurring payments on the lost provider that are on hold, and you are not dying, so you can resolve that problem using a lawyer.

This is crisis management, not technical perfection that you need for those situations.

replies(1): >>32857817 #
11. brightball ◴[] No.32855589[source]
I will definitely look into that.
12. canadiantim ◴[] No.32855597[source]
I'm just starting to use Lago (getlago.com) for this. I'm not affiliated with them, but it solves the need for me for an open-source payment-processor-agnostic billing system that can easily swap between processors while maintaining a single source of truth.
replies(1): >>32863156 #
13. hinkley ◴[] No.32855625{3}[source]
SLAs only entitle you to a refund, not compensation for lost income. If you’re not making at least four times as much off a service as you pay for it, you really need to think about why you use that service.

Power can go out. Promises from the power company don’t fix that. Only backup power does.

replies(1): >>32858151 #
14. canadiantim ◴[] No.32855636[source]
Have you tried or considered lago? (getlago.com) I haven't heard of killbill, so will definitely look into them. Love that they're open-source as well and seems like they've been at it a long while.
replies(3): >>32856060 #>>32856951 #>>32859960 #
15. manquer ◴[] No.32855655[source]
You still have a Single Point of Failure. ChargeBee could shut you down too.

You are right that redundancy is important, but redundancy either in cloud vendors , payment processors or even high availability of your app takes time and effort with no immediate ROI as apposed to buliding features , better customer service.

When running a small business you always take lot of risks by cutting short processes large organisations will have. Judging which ones to take and which to mitigate is a not a easy skill, many times people get it wrong .

16. mardifoufs ◴[] No.32855804[source]
But there are tons of rules and laws around payments already, and they are often the reason why providers are so trigger happy and conservative even if it means losing clients. Regulatory requirements (KYC, money laundering, sanctions) usually force them to make those calls, quickly and by design. It's very clear that most financial regulations are customer/client unfriendly, and inherently treat them with distrust.

I'm not saying that's an inherently good or bad thing... But it sure would be hard to fit both customer protections laws and service guarantees while at the same time having laws that explicitly force providers to do the opposite.

replies(1): >>32863768 #
17. gingerlime ◴[] No.32855987[source]
Besides being a single point of failure, we tried to switch to ChargeBee and found them to be pretty lacking (to put it mildly). The platform looks nice and has lots of options, but under the hood things seem pretty fragile. Horrible docs, random 500 errors that weren't showing on the status page (and support ignored until more customers reported it), our Stripe gateway disconnected and the system still appeared to be working (partly, partly failing) with no alerts, plus lots of other odd behaviour. To be fair, all of this was on the test system, but we didn't feel confident to go live so we put the migration on hold. YMMV of course.
replies(1): >>32858099 #
18. spaceywilly ◴[] No.32855991[source]
Yeah my “business” (just a small app) uses a third party API for tracking packages. You better believe I abstracted that API and programmed to the abstraction, so in case I ever need to move off that API I can do it very easily.

It’s not just getting banned, they could change their pricing on you or just straight up close their doors. You never want your business to be totally dependent on another company. If it can’t be avoided, get on a service contract with them.

19. yamtaddle ◴[] No.32856036{3}[source]

    display_cc_form_that_handles_cc_data_so_we_do_not_have_to() {
        which_one = random_number(3)
        switch which_one {
            case 1:
                display_authorizedotnet_form()
                break
            case 2:
                display_stripe_form()
                break
            default:
                display_paypal_form()
        }
    }

Plus the same integration work for each one that you'd have to do anyway (which may be little or none if you're using a platform that integrates all of them via plugins or settings or whatever)

Like maybe don't literally randomize it request-by-request, but that's how you'd be ready to use multiple processors, and you could do something a little more complex to, say, rotate which one you're on every Wednesday, or whatever. Or just have it ready so a one-line code change or config toggle switches which one you're on (that's only worse because if something's not used frequently in prod, there's a good chance it doesn't actually work, even if it once did)

replies(3): >>32856121 #>>32856296 #>>32857787 #
20. carrotcarrot ◴[] No.32856053[source]
You think stripe and PayPal are not keen to control the flow of information and business success?
21. gingerlime ◴[] No.32856060{3}[source]
Thank you! First I hear about either of those, but definitely interesting. Are there paid / hosted versions of lago? after a sub-par experience trying to switch to ChargeBee I'm looking for alternatives, but struggle to find something that fits our needs.
replies(1): >>32861989 #
22. pc86 ◴[] No.32856121{4}[source]
It's been a long time since I've worked in eCommerce but at my last ecom role we used two payment processors and simply stored the card data with both for every customer, and the relevant IDs in the customer record in the database. Whenever anything got charged we'd check which processor we were supposed to use and off we went.

Yes this has an increased cost if your processor charges by number of customers, but I don't think that's particularly common - these two were just revenue + monthly fee.

replies(1): >>32857047 #
23. PeterisP ◴[] No.32856296{4}[source]
The problem is with recurring payments. If you used this approach and 1/3 of your customers were initially billed through Stripe, then if Stripe bans you then you can't charge the next month's bill through something else because you don't have the stored data for it, you have to ask them to re-enter data in another processor's system.
replies(1): >>32856985 #
24. chromatin ◴[] No.32856951{3}[source]
Lago looks interesting, and promising if development continues, but nowhere near as feature complete or battle tested as Kill Bill.
25. yamtaddle ◴[] No.32856985{5}[source]
Another wrinkle, in practice, is that you can often negotiate significant discounts on fees, based on volume, if you've got enough of it. Yes, including with Stripe and other places that have fairly-transparent public pricing—the big-boys don't pay those rates, they pay lower ones negotiated with a sales rep.

Splitting up your payments reduces your volume with each one, which can mean you're paying higher rates overall. Or, if you go the "keep an unused alternative on standby" route, you'll likely at least have some initial traffic that pays higher public-pricing rates until you can convince them to give you a better rate, and put it in effect.

Still might be worth it as a kind of insurance premium, but it's something to consider.

26. exhilaration ◴[] No.32857047{5}[source]
Great solution, I wouldn't have thought of this. Did you ever get complaints about the charge description looking different month to month, or did both processors pass the same description to the customer's credit card?
replies(1): >>32857674 #
27. pc86 ◴[] No.32857674{6}[source]
I don't remember for sure (this was pushing 10 years ago and early in my career), but I don't recall any complaints of that type.
28. eropple ◴[] No.32857787{4}[source]
This isn't "rotating payment providers", though, this is sharding customers across them. Which may be a good idea, insofar as it could reduce your blast radius, but it doesn't allow for portable customers--which seemed to be what the GGP was implying.

'pc86 has an interesting solution in a sibling comment, but I don't think you can do this across the high-touch providers, eg Stripe+Paypal.

29. ◴[] No.32857817{4}[source]
30. brightball ◴[] No.32858099{3}[source]
Fwiw, I’m just using them as an example. There are likely plenty of other options.
31. brightball ◴[] No.32858151{4}[source]
Point here was that you’d hypothetically be able to get all of your data to setup an alternative if you had to, rather than just being shut down and waiting for them to fix it.
replies(1): >>32868379 #
32. connordoner ◴[] No.32859526{3}[source]
The problem with that idea is that PCI compliance doesn't just include the software you're running, but the infrastructure you run it on, various elements of organisational security and, of course, certification costs.
33. NetOpWibby ◴[] No.32859960{3}[source]
Lago looks beautiful, thanks for sharing!
34. canadiantim ◴[] No.32861989{4}[source]
Yep there is a paid / hosted version of lago; I believe you can book a demo of the hosted version from their website and get a good 1 on 1 with one of their developers so you can see if it works for your needs.
35. cx42net ◴[] No.32863156[source]
But what happens if getlago.com bans you? Haven't you just moved the single-point-of-failure at a different place?
replies(1): >>32867176 #
36. jstummbillig ◴[] No.32863768{3}[source]
There is nothing intrinsic about financial regulations that needs to be customer unfriendly. Checks can work both ways and what happened at stripe could for example be avoided by a statue which requires a reasonable explanation and a clear path and timeline towards resolution for the customer, regardless of the company they work with.
37. KaoruAoiShiho ◴[] No.32865985[source]
That's why you need a self hosted solution.
38. canadiantim ◴[] No.32867176{3}[source]
It's open source, so I can just set up my own instance. I just have to be careful to make sure that process is as easy as possible. You never know what provider may ban you so yeah that's definitely a concern, but that's why I build on only open source.
replies(1): >>32896042 #
39. hinkley ◴[] No.32868379{5}[source]
Once you trust your vendors absolutely, your company becomes their company, because they can do whatever they want with it.

What always confuses me about this is that it seems like many owners trust their vendors over their own employees and I don't even know where to start unpacking that sentence.

40. cx42net ◴[] No.32896042{4}[source]
Oh wow, I thought it was a paid service! That is good! Thank you for the clarification !