←back to thread

Fly.io Postgres cluster down for 3 days, no word from them about it

(webcache.googleusercontent.com)
797 points burnerbob | 4 comments | 20 Jul 23 23:42 UTC | HN request time: 0s | source
Show context
tptacek ◴[21 Jul 23 05:33 UTC] No.36810326[source]▶
Y'all, this is going to be deeply unsatisfying, but it's what I can report personally:

I have no earthly clue why this thread on our community site is unlisted.

We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread (the site is public, anybody can sign up for an account and see the thread).

Say what you want about us, but hiding out from stuff like this isn't one of our flaws. When I find out more about what happened with this thread, I'll let you know (or Kurt will reply here and tell me I'm wrong).

I don't know enough about what happened with this Sydney server to be helpful to people who had instances running on it. When I know more about it, I'll be helpful, but I'm just learning about this stuff right now, after getting back in from a night out.

Almost immediately afterwards

It looks like... all the posts in the app-not-working category are "private"? Like it's some setting on the category itself? "Private" here means you need to have signed up for a Discourse account to see them?

replies(12): >>36810339 #>>36810345 #>>36810393 #>>36810467 #>>36810497 #>>36810498 #>>36810755 #>>36810983 #>>36812367 #>>36812723 #>>36812856 #>>36834726 #
xupybd ◴[21 Jul 23 06:04 UTC] No.36810497[source]▶
From this my take away is that I could get fired for picking Fly.io for work. Not because there was an outage but because days could pass before getting support.

What assurances could you give the community here that the support would be better next time?

replies(4): >>36810530 #>>36810775 #>>36811112 #>>36812013 #
tinco ◴[21 Jul 23 06:53 UTC] No.36810775[source]▶
Try filing a bug with any of the big three cloud vendors when you're on their free plan. It's really not different, the thing that is going to get you fired is not realizing you're not paying a couple hundred bucks per month for premium service on the infrastructure that is mission critical to your company.
replies(2): >>36810915 #>>36814436 #
1. xupybd ◴[21 Jul 23 07:15 UTC] No.36810915[source]▶
Funny story, when I started my current role I researched our hosting provider. I couldn't find the matching invoices in the accounting system. So I called the vendor, a local company. They'd not set our account up correctly, billing was not enabled. Since then we've been billed. I'm glad we sorted it but it wasn't a good look to start my role by increasing our spending.
replies(2): >>36811268 #>>36811427 #
2. CameronNemo ◴[21 Jul 23 08:04 UTC] No.36811268[source]▶
My neighbor once had a gardener who delivered no bill. For years! Then out of the blue, $4k invoice.

Trust me, you did the business a favor.

3. mst ◴[21 Jul 23 08:29 UTC] No.36811427[source]▶
I feel like starting your role by discovering a crucial service wasn't being paid for and therefore was at risk of suddenly going away should be a pretty positive thing.

However 'should' is pretty load bearing there and actual results are probably heavily dependent on management culture and the current state of office politics.

replies(1): >>36811920 #
4. tinco ◴[21 Jul 23 09:47 UTC] No.36811920[source]▶
We had a customer once that our automatic billing system tried to reach for 3 months about failing credit card charges (<$5k/mo). Our system stopped the service.. I'm pretty sure their subsequent outage cost their customers millions. Lessons about what it means to have (and be) enterprise customers were learned. Unfortunately the lady who was ignoring our e-mails in her inbox got fired.