←back to thread

Fly.io Postgres cluster down for 3 days, no word from them about it

(webcache.googleusercontent.com)
797 points burnerbob | 4 comments | 20 Jul 23 23:42 UTC | HN request time: 1.009s | source
Show context
spiderice ◴[21 Jul 23 03:25 UTC] No.36809650[source]▶
There is now a response to the support thread from Fly[1]:

> Hi Folks,

> Just wanted to provide some more details on what happened here, both with the thread and the host issue.

> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

[1] https://community.fly.io/t/service-interruption-cant-destroy...

replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
gowthamgts12 ◴[21 Jul 23 03:34 UTC] No.36809693[source]▶
> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally.

If it really got missed, then I don't understand how the thread was made private to only logged-in users?

replies(3): >>36810248 #>>36810251 #>>36810285 #
teraflop ◴[21 Jul 23 05:23 UTC] No.36810285[source]▶
It looks like all 166 threads with the "App not working" tag are invisible when not logged in. So I'm guessing somebody applied that tag retroactively.

https://community.fly.io/c/questions-and-help/app-not-workin...

EDIT: it now appears that the "app-not-working" tag itself has been deleted, and no longer shows up even when logged in.

replies(2): >>36810603 #>>36810620 #
1. buro9 ◴[21 Jul 23 06:22 UTC] No.36810603[source]▶
This is why companies should not run their own forums. It's cheap support and marketing, it's not really community.
replies(1): >>36810656 #
2. vasco ◴[21 Jul 23 06:31 UTC] No.36810656[source]▶
I never thought to make friends with people who's only common thing with me is that they shop at the same place. Companies creating a "community" is exactly as you described.
replies(1): >>36811396 #
3. toyg ◴[21 Jul 23 08:23 UTC] No.36811396[source]▶
I am an interested party in the process space, and I think that's ungenerous. When you work with a complex tool every day, and you have to find solutions for this or that issue, develop strategies for this or that business case, etc etc, you're not really shopping - it's more like you're in the trenches. At that point, finding people who have the same issues and talking shop with them, can be great for both knowledge exchange and camaraderie. Linux wouldn't be what it is today without the LUGs era, for example.
replies(1): >>36811870 #
4. vasco ◴[21 Jul 23 09:40 UTC] No.36811870{3}[source]▶
We're talking about private companies running forum software instead of providing support. We're not talking about the power of IRC or mailing list communities for open source projects and the like.

If I pay for something I want the person I pay money to help me fix problems I get.