←back to thread

797 points burnerbob | 9 comments | | HN request time: 1.074s | source | bottom
Show context
spiderice ◴[] No.36809650[source]
There is now a response to the support thread from Fly[1]:

> Hi Folks,

> Just wanted to provide some more details on what happened here, both with the thread and the host issue.

> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

[1] https://community.fly.io/t/service-interruption-cant-destroy...

replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
gowthamgts12 ◴[] No.36809693[source]
> While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally.

If it really got missed, then I don't understand how the thread was made private to only logged-in users?

replies(3): >>36810248 #>>36810251 #>>36810285 #
p-e-w ◴[] No.36810251[source]
Whoa, what? That's a much bigger red flag than the downtime itself.
replies(1): >>36810657 #
1. throwawayfly ◴[] No.36810657[source]
Ok as long as we’re getting conspiratorial, something similar I observed has bugged me.

About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me.

One person I saw with it absolutely deserved it: this person is, to this day, always hopping in and helping people; linking to docs; raising their own issues with a big dose of “fellow builder” understanding and empathy; that sort of person. My own queries typically led me to a thread that this person has answered. In short - the kind of helpful, proactive, high knowledge volunteer early adopter that every community needs - and a handful are blessed to find.

Then one day I saw this same person had offered — to one random newbie with build problems in one of the many HALP threads — a reply like, “maybe Fly isn’t the best option for you. here are some other places that can host an app”.

The thread was left alone and faded, like many when a lost newbie is involved. But 1 day later, I noticed this tireless early adopter no longer had their “aeronaut” badge.

I still refuse to believe my own eyes about something that petty.

replies(2): >>36810885 #>>36810935 #
2. michaeldwan ◴[] No.36810885[source]
Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same?

Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut

replies(4): >>36811286 #>>36812821 #>>36812879 #>>36813245 #
3. logeist ◴[] No.36810935[source]
Conspiratorial or not that's enough for me to never use it. God forbid someone recommends another platform that handles your clear shortcomings.
replies(1): >>36811424 #
4. sho ◴[] No.36811286[source]
> Do you really think we have a desire (or time) to punish somebody for doing the same?

idk man, there's these awfully convenient disappearing forum threads too. The benefit of the doubt is starting to expire.

I see you're a co-founder, so presumably you have some sway on priorities and skin in the game. I think you should take the reputational damage you're accruing here much more seriously than you apparently are. A few more incidents like this and it won't just be you telling people you're a bad option.

* edited to tone down the forum thread disappearance angle. FWIW I do believe that it likely wasn't deliberate. My main point was that these things add up and "of course we wouldn't do that!" starts to ring a little hollow the 10th time you hear it...

replies(1): >>36811385 #
5. p-e-w ◴[] No.36811385{3}[source]
> you've just been caught hiding inconvenient forum threads too

FWIW, I do believe them when they say this wasn't intentional. Considering how the Internet operates, they would be incredibly stupid to do something like that on purpose.

That being said, the way the entire affair was handled certainly leaves a lot to be desired.

replies(1): >>36811496 #
6. robertlagrant ◴[] No.36811424[source]
> Conspiratorial or not that's enough for me to never use it

Well if it's not true then that would be a silly reason to pick to not use them.

7. sho ◴[] No.36811496{4}[source]
I actually believe them on that too, FWIW. This time. It's just too dumb. I hope, for their sake, it's the truth.

I was really just trying to point out that this kind of good faith benefit-of-the-doubt has a limit, and fear of reaching that limit should be keeping people at fly up at night a lot more than it apparently is. I don't know how many colossal public fuckups a company can endure before its reputation is permanently ruined, but it's definitely not infinite.

8. kotaKat ◴[] No.36812821[source]
Why are you acting so hostile? If you don't like that the community is dunking on you, then maybe posting on Hacker News isn't for you.
9. ryanrussell ◴[] No.36813245[source]
Why is anyone on HN "dunking" on Fly.IO of all companies?

Michael - Don't take the bait.

As someone who has zero affiliation with Fly.IO other than a few PR's to their OSS(I don't even know Michael), I greatly appreciate the contributions they have given back to the community.

There are a lot of great hosting companies. Fly.IO stands out due to their revolutionary architecture and contributions back to the OSS community. I wish more companies operated like this.

It's understandable some are upset about an outage. But Fly is doing really interesting and game-changing things, not copying a traditional vmware, cpanel or k8s route.

Just as a reminder to what this company has offered back to everyone.

SQLite: Ben Johnson's OSS work around SQLite stands out. Fly.IO and his work have really made sqlite a contender. - https://fly.io/blog/all-in-on-sqlite-litestream/ - https://fly.io/blog/introducing-litefs/ - https://github.com/superfly/litefs - https://github.com/benbjohnson/litestream - https://fly.io/blog/sqlite-internals-wal/ - https://fly.io/blog/wal-mode-in-litefs/

Who really considered sqlite as a production option before Fly and Ben? Not me.

Firecracker: Firecracker is amazing, but difficult to debug when something bad happens. There aren't a ton of people in devops who would share what they have. If you've ever used Firecracker, you've really been helped a lot by the various guides they have provided back to the community like these: - https://fly.io/docs/reference/architecture/ - https://fly.io/blog/fly-machines/ - https://fly.io/blog/sandboxing-and-workload-isolation/

Their architecture is beautiful and revolutionary. They're probably the first or second ones to find a lot of the new edge cases as they grow.

It's a lot harder to be the first one over the wall than it is to copy. They've literally given the average developer a blueprint to build scalable businesses that compete with their own.