←back to thread

Fly.io Postgres cluster down for 3 days, no word from them about it

(webcache.googleusercontent.com)
797 points burnerbob | 9 comments | 20 Jul 23 23:42 UTC | HN request time: 0.291s | source | bottom
Show context
spiderice ◴[21 Jul 23 03:25 UTC] No.36809650[source]▶
There is now a response to the support thread from Fly[1]:

> Hi Folks,

> Just wanted to provide some more details on what happened here, both with the thread and the host issue.

> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

[1] https://community.fly.io/t/service-interruption-cant-destroy...

replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
mrcwinn ◴[21 Jul 23 03:39 UTC] No.36809725[source]▶
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
replies(5): >>36809880 #>>36810018 #>>36810039 #>>36810724 #>>36814012 #
steve_adams_86 ◴[21 Jul 23 04:07 UTC] No.36809880[source]▶
I left digitalocean for fly because some of their tooling was excellent. I was pretty excited.

I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers.

I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them.

It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.

replies(4): >>36810127 #>>36810379 #>>36813660 #>>36813890 #
danielvaughn ◴[21 Jul 23 04:53 UTC] No.36810127[source]▶
I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.
replies(10): >>36810554 #>>36810628 #>>36810638 #>>36812302 #>>36813142 #>>36813668 #>>36814283 #>>36823458 #>>36827607 #>>36834710 #
1. JanSt ◴[21 Jul 23 06:26 UTC] No.36810628[source]▶
I'm using Digital Ocean App platform, which does pretty much everything for me. It's very simple to use. I can run my app as a single developer without caring about infrastructure for 99% of the time.
replies(2): >>36811443 #>>36823542 #
2. fauigerzigerk ◴[21 Jul 23 08:32 UTC] No.36811443[source]▶
Do they offer authentication/authorization?

This is the one thing I need in every app and don't want to do myself.

replies(4): >>36811797 #>>36813386 #>>36813744 #>>36814722 #
3. spacebanana7 ◴[21 Jul 23 09:29 UTC] No.36811797[source]▶
I've been using Supabase for authentication/authorization in my recent side project.

The main app is node/express running on Digital Ocean and it connects to directly to the Supabase hosted Postgres for most operations, but then uses the Supabase auth API for auth related stuff.

Saves a lot of time sending password reset emails etc and the entire project costs less than $5/mo in hosting costs.

4. okhuman ◴[21 Jul 23 13:16 UTC] No.36813386[source]▶
Would you consider a project like https://github.com/authcompanion/authcompanion2 for the authentication side? Missing anything?
replies(1): >>36814350 #
5. pc86 ◴[21 Jul 23 13:50 UTC] No.36813744[source]▶
In addition to Supabase Auth the sibling mentions (which I played with very briefly) I've been using clerk.dev (no affiliation) and it's great. Depending on your definition of doing it yourself it could be just want you want. You have to set some things up, you're not going to get things like row-level permissions you get out of the box w/ Supabase, but if you're looking for a quick implementation where things like password reset etc. are handled for you, it might be a good fit.
6. fauigerzigerk ◴[21 Jul 23 14:39 UTC] No.36814350{3}[source]▶
No I would not.

I don't like self hosting anything that requires its own process. And if I did decide to self host I would choose a more mature project.

This is a very young one man project delegating the heavy lifting to another one man project. And it doesn't appear to support social logins.

replies(1): >>36819868 #
7. LtdJorge ◴[21 Jul 23 15:08 UTC] No.36814722[source]▶
I like https://github.com/goauthentik It has Helm charts and a Terraform provider.
8. okhuman ◴[21 Jul 23 21:13 UTC] No.36819868{4}[source]▶
thanks for the feedback.
9. steve_adams_86 ◴[22 Jul 23 05:24 UTC] No.36823542[source]▶
Same, it works really well.

Part of what inspired me to give fly.io a shot was that I didn’t love the monorepo deployment story on the app platform. Fly doesn’t have a solution to that, but I suppose I felt less tied to DO at the time because I wasn’t totally content anyways. I’ve discovered since then that I was actually doing it wrong, so I’m way happier. I’m pretty big on monorepos so their whole system fits my workflow remarkably well now.

I’d like to figure out how to prevent deployments when my code doesn’t change in one app, but does in another. At the moment, pushing anything at all will trigger all apps to rebuild and deploy again. Not a huge deal and several orders of magnitude less painful than not being able to deploy at all, haha.