←back to thread

797 points burnerbob | 10 comments | | HN request time: 0.001s | source | bottom
Show context
spiderice ◴[] No.36809650[source]
There is now a response to the support thread from Fly[1]:

> Hi Folks,

> Just wanted to provide some more details on what happened here, both with the thread and the host issue.

> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

[1] https://community.fly.io/t/service-interruption-cant-destroy...

replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
mrcwinn ◴[] No.36809725[source]
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
replies(5): >>36809880 #>>36810018 #>>36810039 #>>36810724 #>>36814012 #
steve_adams_86 ◴[] No.36809880[source]
I left digitalocean for fly because some of their tooling was excellent. I was pretty excited.

I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers.

I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them.

It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.

replies(4): >>36810127 #>>36810379 #>>36813660 #>>36813890 #
NicoJuicy ◴[] No.36810379[source]
I moved from DO to Hetzner ( cheaper), I am happy about it.
replies(7): >>36810595 #>>36810697 #>>36810760 #>>36810809 #>>36810954 #>>36812172 #>>36813077 #
YetAnotherNick ◴[] No.36810760[source]
Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?
replies(6): >>36810793 #>>36811511 #>>36811571 #>>36811650 #>>36812116 #>>36812917 #
1. stevefan1999 ◴[] No.36810793{4}[source]
Simple, Hetzner mainly operates on Germany, the people are mostly Germans, and they automate the stuff to a point a small team could manage it well even if not remotely, so they have less cost on human resources.
replies(3): >>36811063 #>>36812572 #>>36813431 #
2. rahkiin ◴[] No.36811063[source]
They also build their own servers in their own datacenters
replies(1): >>36811239 #
3. raybb ◴[] No.36811239[source]
Does digital ocean not do this?
replies(3): >>36811553 #>>36811969 #>>36812259 #
4. re-thc ◴[] No.36811553{3}[source]
They don’t.
replies(1): >>36814363 #
5. ◴[] No.36811969{3}[source]
6. stevefan1999 ◴[] No.36812259{3}[source]
The competitor of DO, Vultr does this IIRC, yet it is not really cheaper
7. KronisLV ◴[] No.36812572[source]
> Simple, Hetzner mainly operates on Germany, the people are mostly Germans, and they automate the stuff to a point a small team could manage it well even if not remotely, so they have less cost on human resources.

I feel like there might be more to it, especially considering the situation with electricity prices in some places in EU recently.

I used (and still use) a Lithuanian platform called Time4VPS which was cheaper than Hetzner previously, yet had to increase their prices somewhat for that reason. Now only some of their plans are competitive with Hetzner, while Hetzner also provides some managed services as well.

Hetzner docs also went into some of the details regarding the pricing: https://docs.hetzner.com/robot/general/pricing/hetzner-prici...

And yet, I can't help but to wonder why they don't give in to the desire to maximize profit margins, like happened to say Scaleway (good platform, but as expensive as DigitalOcean).

8. api ◴[] No.36813431[source]
I've wondered how they can host this cheap in Germany given their very high electricity prices.

Maybe that's not actually the dominant cost, or they've optimized everything else so well they can just eat the electric bill.

9. cutemonster ◴[] No.36814363{4}[source]
Where do DO get their servers and data centers from? ... Apparently they run on AWS, I'm surprised
replies(1): >>36834390 #
10. re-thc ◴[] No.36834390{5}[source]
> Apparently they run on AWS, I'm surprised

They don't run on AWS. Not sure what sort of rumors are running :(

> data centers from?

The major players e.g. Equinix, Coresite, etc. Varies per location. Even AWS don't build most of their data centers.