Most active commenters

    ←back to thread

    Fly.io Postgres cluster down for 3 days, no word from them about it

    (webcache.googleusercontent.com)
    797 points burnerbob | 19 comments | 20 Jul 23 23:42 UTC | HN request time: 0.944s | source | bottom
    Show context
    spiderice ◴[21 Jul 23 03:25 UTC] No.36809650[source]▶
    There is now a response to the support thread from Fly[1]:

    > Hi Folks,

    > Just wanted to provide some more details on what happened here, both with the thread and the host issue.

    > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

    > More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

    > Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

    > The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

    > It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

    [1] https://community.fly.io/t/service-interruption-cant-destroy...

    replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
    1. gowthamgts12 ◴[21 Jul 23 03:34 UTC] No.36809693[source]▶
    > While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally.

    If it really got missed, then I don't understand how the thread was made private to only logged-in users?

    replies(3): >>36810248 #>>36810251 #>>36810285 #
    2. ◴[21 Jul 23 05:17 UTC] No.36810248[source]▶
    3. p-e-w ◴[21 Jul 23 05:17 UTC] No.36810251[source]▶
    Whoa, what? That's a much bigger red flag than the downtime itself.
    replies(1): >>36810657 #
    4. teraflop ◴[21 Jul 23 05:23 UTC] No.36810285[source]▶
    It looks like all 166 threads with the "App not working" tag are invisible when not logged in. So I'm guessing somebody applied that tag retroactively.

    https://community.fly.io/c/questions-and-help/app-not-workin...

    EDIT: it now appears that the "app-not-working" tag itself has been deleted, and no longer shows up even when logged in.

    replies(2): >>36810603 #>>36810620 #
    5. buro9 ◴[21 Jul 23 06:22 UTC] No.36810603[source]▶
    This is why companies should not run their own forums. It's cheap support and marketing, it's not really community.
    replies(1): >>36810656 #
    6. kipple ◴[21 Jul 23 06:25 UTC] No.36810620[source]▶
    In another comment here, they're saying they just deleted that tag to avoid this access issue — https://news.ycombinator.com/item?id=36810393
    replies(1): >>36810752 #
    7. vasco ◴[21 Jul 23 06:31 UTC] No.36810656{3}[source]▶
    I never thought to make friends with people who's only common thing with me is that they shop at the same place. Companies creating a "community" is exactly as you described.
    replies(1): >>36811396 #
    8. throwawayfly ◴[21 Jul 23 06:31 UTC] No.36810657[source]▶
    Ok as long as we’re getting conspiratorial, something similar I observed has bugged me.

    About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me.

    One person I saw with it absolutely deserved it: this person is, to this day, always hopping in and helping people; linking to docs; raising their own issues with a big dose of “fellow builder” understanding and empathy; that sort of person. My own queries typically led me to a thread that this person has answered. In short - the kind of helpful, proactive, high knowledge volunteer early adopter that every community needs - and a handful are blessed to find.

    Then one day I saw this same person had offered — to one random newbie with build problems in one of the many HALP threads — a reply like, “maybe Fly isn’t the best option for you. here are some other places that can host an app”.

    The thread was left alone and faded, like many when a lost newbie is involved. But 1 day later, I noticed this tireless early adopter no longer had their “aeronaut” badge.

    I still refuse to believe my own eyes about something that petty.

    replies(2): >>36810885 #>>36810935 #
    9. swyx ◴[21 Jul 23 06:48 UTC] No.36810752{3}[source]▶
    good call out - please as an internet mob let us not ascribe to malice what can be attributed to sheer unintentional impacts of complex software
    10. michaeldwan ◴[21 Jul 23 07:12 UTC] No.36810885{3}[source]▶
    Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same?

    Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut

    replies(4): >>36811286 #>>36812821 #>>36812879 #>>36813245 #
    11. logeist ◴[21 Jul 23 07:17 UTC] No.36810935{3}[source]▶
    Conspiratorial or not that's enough for me to never use it. God forbid someone recommends another platform that handles your clear shortcomings.
    replies(1): >>36811424 #
    12. sho ◴[21 Jul 23 08:07 UTC] No.36811286{4}[source]▶
    > Do you really think we have a desire (or time) to punish somebody for doing the same?

    idk man, there's these awfully convenient disappearing forum threads too. The benefit of the doubt is starting to expire.

    I see you're a co-founder, so presumably you have some sway on priorities and skin in the game. I think you should take the reputational damage you're accruing here much more seriously than you apparently are. A few more incidents like this and it won't just be you telling people you're a bad option.

    * edited to tone down the forum thread disappearance angle. FWIW I do believe that it likely wasn't deliberate. My main point was that these things add up and "of course we wouldn't do that!" starts to ring a little hollow the 10th time you hear it...

    replies(1): >>36811385 #
    13. p-e-w ◴[21 Jul 23 08:21 UTC] No.36811385{5}[source]▶
    > you've just been caught hiding inconvenient forum threads too

    FWIW, I do believe them when they say this wasn't intentional. Considering how the Internet operates, they would be incredibly stupid to do something like that on purpose.

    That being said, the way the entire affair was handled certainly leaves a lot to be desired.

    replies(1): >>36811496 #
    14. toyg ◴[21 Jul 23 08:23 UTC] No.36811396{4}[source]▶
    I am an interested party in the process space, and I think that's ungenerous. When you work with a complex tool every day, and you have to find solutions for this or that issue, develop strategies for this or that business case, etc etc, you're not really shopping - it's more like you're in the trenches. At that point, finding people who have the same issues and talking shop with them, can be great for both knowledge exchange and camaraderie. Linux wouldn't be what it is today without the LUGs era, for example.
    replies(1): >>36811870 #
    15. robertlagrant ◴[21 Jul 23 08:29 UTC] No.36811424{4}[source]▶
    > Conspiratorial or not that's enough for me to never use it

    Well if it's not true then that would be a silly reason to pick to not use them.

    16. sho ◴[21 Jul 23 08:41 UTC] No.36811496{6}[source]▶
    I actually believe them on that too, FWIW. This time. It's just too dumb. I hope, for their sake, it's the truth.

    I was really just trying to point out that this kind of good faith benefit-of-the-doubt has a limit, and fear of reaching that limit should be keeping people at fly up at night a lot more than it apparently is. I don't know how many colossal public fuckups a company can endure before its reputation is permanently ruined, but it's definitely not infinite.

    17. vasco ◴[21 Jul 23 09:40 UTC] No.36811870{5}[source]▶
    We're talking about private companies running forum software instead of providing support. We're not talking about the power of IRC or mailing list communities for open source projects and the like.

    If I pay for something I want the person I pay money to help me fix problems I get.

    18. kotaKat ◴[21 Jul 23 12:00 UTC] No.36812821{4}[source]▶
    Why are you acting so hostile? If you don't like that the community is dunking on you, then maybe posting on Hacker News isn't for you.
    19. ryanrussell ◴[21 Jul 23 13:00 UTC] No.36813245{4}[source]▶
    Why is anyone on HN "dunking" on Fly.IO of all companies?

    Michael - Don't take the bait.

    As someone who has zero affiliation with Fly.IO other than a few PR's to their OSS(I don't even know Michael), I greatly appreciate the contributions they have given back to the community.

    There are a lot of great hosting companies. Fly.IO stands out due to their revolutionary architecture and contributions back to the OSS community. I wish more companies operated like this.

    It's understandable some are upset about an outage. But Fly is doing really interesting and game-changing things, not copying a traditional vmware, cpanel or k8s route.

    Just as a reminder to what this company has offered back to everyone.

    SQLite: Ben Johnson's OSS work around SQLite stands out. Fly.IO and his work have really made sqlite a contender. - https://fly.io/blog/all-in-on-sqlite-litestream/ - https://fly.io/blog/introducing-litefs/ - https://github.com/superfly/litefs - https://github.com/benbjohnson/litestream - https://fly.io/blog/sqlite-internals-wal/ - https://fly.io/blog/wal-mode-in-litefs/

    Who really considered sqlite as a production option before Fly and Ben? Not me.

    Firecracker: Firecracker is amazing, but difficult to debug when something bad happens. There aren't a ton of people in devops who would share what they have. If you've ever used Firecracker, you've really been helped a lot by the various guides they have provided back to the community like these: - https://fly.io/docs/reference/architecture/ - https://fly.io/blog/fly-machines/ - https://fly.io/blog/sandboxing-and-workload-isolation/

    Their architecture is beautiful and revolutionary. They're probably the first or second ones to find a lot of the new edge cases as they grow.

    It's a lot harder to be the first one over the wall than it is to copy. They've literally given the average developer a blueprint to build scalable businesses that compete with their own.