Most active commenters
  • ushakov(3)
  • abwizz(3)
  • dpeckett(3)

←back to thread

797 points burnerbob | 48 comments | | HN request time: 0.001s | source | bottom
Show context
spiderice ◴[] No.36809650[source]
There is now a response to the support thread from Fly[1]:

> Hi Folks,

> Just wanted to provide some more details on what happened here, both with the thread and the host issue.

> The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flagged it internally. If we’d seen it earlier, we’d have offered more details the.

> More on what happened: We had a single host in the syd region go down, hard, with multiple issues. In short, the host required a restart, then refused to come back online cleanly. Once back online, it refused to connect with our service discovery system. Ultimately it required a significant amount of manual work to recover.

> Apps running multiple instances would have seen the instance on this host go unreachable, but other instances would have remained up and new instances could be added. Single instance apps on this host were unreachable for the duration of the outage. We strongly recommend running multiple instances to mitigate the impact of single-host failures like this.

> The main status page (status.fly.io) is used for global and regional outages. For single host issues like this one we post alerts on the status tab in the dashboard (the emergency maintenance message @south-paw posted). This was an abnormally long single-host failure and we’re reassessing how these longer-lasting single-host outages are communicated.

> It sucks to feel ignored when you’re having issues, even when it’s not intentional. Sorry we didn’t catch this thread sooner.

[1] https://community.fly.io/t/service-interruption-cant-destroy...

replies(10): >>36809693 #>>36809725 #>>36809824 #>>36809928 #>>36810269 #>>36810740 #>>36811025 #>>36812597 #>>36812956 #>>36813681 #
mrcwinn ◴[] No.36809725[source]
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
replies(5): >>36809880 #>>36810018 #>>36810039 #>>36810724 #>>36814012 #
steve_adams_86 ◴[] No.36809880[source]
I left digitalocean for fly because some of their tooling was excellent. I was pretty excited.

I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers.

I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them.

It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.

replies(4): >>36810127 #>>36810379 #>>36813660 #>>36813890 #
1. NicoJuicy ◴[] No.36810379[source]
I moved from DO to Hetzner ( cheaper), I am happy about it.
replies(7): >>36810595 #>>36810697 #>>36810760 #>>36810809 #>>36810954 #>>36812172 #>>36813077 #
2. x86hacker1010 ◴[] No.36810595[source]
Same here
3. brylie ◴[] No.36810697[source]
I'm enjoying the DO App Platform (Heroku alternative). Do you know if Hetzner has a similar service that I could compare?
replies(2): >>36810730 #>>36857753 #
4. realusername ◴[] No.36810730[source]
Personally I just install Dokku onto the machine, it replaced all my Heroku (and competitors) uses.

Additionally, you still keep the full ssh access to the machine if you ever need it.

5. YetAnotherNick ◴[] No.36810760[source]
Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?
replies(6): >>36810793 #>>36811511 #>>36811571 #>>36811650 #>>36812116 #>>36812917 #
6. stevefan1999 ◴[] No.36810793[source]
Simple, Hetzner mainly operates on Germany, the people are mostly Germans, and they automate the stuff to a point a small team could manage it well even if not remotely, so they have less cost on human resources.
replies(3): >>36811063 #>>36812572 #>>36813431 #
7. mythz ◴[] No.36810809[source]
Same, been enjoying Hetzner's great value for 10 years, and now Hetzner Cloud for 2 years.
8. tacker2000 ◴[] No.36810954[source]
I use both and am very satisfied, especially by Hetzner.
replies(4): >>36811042 #>>36811227 #>>36811651 #>>36857655 #
9. candiddevmike ◴[] No.36811042[source]
Only complaint with Hetzner is they don't have some kind of OAuth setup for machines or scoped API tokens, just read/write. I'd like to use the former for doing Vault authentication from instances, and the latter for writing a dynamic Vault secret provider.
replies(1): >>36812679 #
10. rahkiin ◴[] No.36811063{3}[source]
They also build their own servers in their own datacenters
replies(1): >>36811239 #
11. throw382642 ◴[] No.36811227[source]
I remember someone complaining they had to send Hetzner a passport or some other type of ID to cancel their services.

Does anyone know if that's still the case?

replies(2): >>36811419 #>>36811723 #
12. raybb ◴[] No.36811239{4}[source]
Does digital ocean not do this?
replies(3): >>36811553 #>>36811969 #>>36812259 #
13. selectnull ◴[] No.36811419{3}[source]
They require passport or some sort of ID on registration, and it is weird when compared to others. I was not happy with that part, but I am happy customer since (almost a decade now).

As far as I know, they do not require any ID when canceling the service.

14. ushakov ◴[] No.36811511[source]
Me and my partner have paid a visit to their datacenter in Nüremberg. The answer is efficiency. They get more processing power than the other providers for the energy they have to put in
replies(1): >>36811622 #
15. re-thc ◴[] No.36811553{5}[source]
They don’t.
replies(1): >>36814363 #
16. ushakov ◴[] No.36811571[source]
Efficiency. They get much more processing power per kw/h of energy than everybode else
17. arrowsmith ◴[] No.36811622{3}[source]
What do they do that makes them more efficient?
replies(1): >>36811798 #
18. devjab ◴[] No.36811650[source]
They run their own data centres and have for a while. There is a pretty big industry for that sort of thing as an alternative to “the cloud” here in Europe.

We used to use nianet to house our hardware in Denmark. Basically these companies does hardware renting and they also do hardware renting with more steps which is where you rent rack space but own the hardware. They provide the place for the hardware and they also have multiple locations so that you have both backup and redundancy, and while it doesn’t scale globally in 20 years I’ve literally never worked on anything that needed to beyond having some buffer caches for clients logging in on their vacations or something like that.

What Hetzner seems to be doing with the DO styled hosting, and this is just a guess, is that they are one or the many EU companies preparing for the big EU exodus from the non-EU cloud. Which is frankly a solid bet these days where both AWS and Azure are increasing prices and are becoming more and more unusable because of EU legislation. Part of this is privacy which Microsoft and Amazon are great with in terms of compliance, but part of it is also national security. I work in an investment bank that builds solar plants, since finance and energy are both critical sectors we risk being told that half of the finance/energy companies in the world can’t use Microsoft because the EU seems it as a single point of failure if our entire energy sector relies on Azure. Which is sort of reasonable right? But what this means for us is that we can’t vendor lock-in, not really, because we need to have up-to-date exit strategies for how we plan on being fully operation a month after leaving Azure. Which is easy when you just containerise everything and run it in VMs or similar, and really annoying if you go full in on things like AKS. Which doesn’t help our Azure costs.

Anyway, right now we are planning on leaving Azure because of cost. Not today, not next week but sometime in the next 5-10 years and a lot of these EU cloud alternatives that actually operate the hardware instead of renting it are likely going to be a very realistic alternative. And that is the private sector, I spend time in the EU public sector which is a massive amount of money and I’m guessing it’ll leave both AWS and Azure by 2050. Some of these EU cloud initiatives is going to explode when that happens, and right now, hetzner is one of the best bets.

To get back to your question, DO rents server space. I have no idea where they’d rent it in Germany but they could potentially be renting it from Hetzner.

replies(3): >>36811770 #>>36811838 #>>36813024 #
19. kristiandupont ◴[] No.36811651[source]
Do they have Terraform providers? And managed Postgres? Besides from the ability to just host a Docker container, that is all I need.
replies(1): >>36811955 #
20. fx1994 ◴[] No.36811723{3}[source]
Well I would appreciate that, since I was victim of russian hackers and they had access to all my servers and stuff on Hetzer, they even changed passwords and mail on Robot but i restored everything...
21. abwizz ◴[] No.36811770{3}[source]
commendable to plan a few years ahead, but betting on the state of cloud business 26years from now seems a bit over the top
replies(2): >>36812142 #>>36815807 #
22. abwizz ◴[] No.36811798{4}[source]
i'll guess they pick optimized components for it.

like the longtime workhorse was a high performance skylake desktop cpu w/o ecc ram

replies(1): >>36813464 #
23. dpeckett ◴[] No.36811838{3}[source]
Couldn't agree more, I think Hetzner is probably Europe's best bet on a hyperscaler. One of the more telling indicators IMO is their growing market share outside of the EU/DACH.

To add on to the comments about Hetzner building their own custom hardware, they also custom built their own software stack. They rejected the hype that was OpenStack and worked diligently on their own hypervisor platform (that they are incredibly secretive about) and that appears to be paying off in spades for them. Most sovereign cloud plays end up being suffocated by the complexity, and incoherence, of the OpenStack ecosystem. It just becomes impossible to ship.

For a fascinatingly different take on how to build a datacenter: https://www.youtube.com/watch?v=5eo8nz_niiM

* Edit: remove speculation about Kubernetes and Hetzner, that was based on hazy memory.

replies(2): >>36812139 #>>36812449 #
24. d_k_f ◴[] No.36811955{3}[source]
Yes and (unfortunately) no. Terraform providers are here [1] with the official documentation at [2]. Managed databases are not available, though. I think they have some sort of database offering if you select their web hosting options, but you can't just get a managed Postgres instance yourself.

[1] https://registry.terraform.io/providers/hetznercloud/hcloud/... [2] https://community.hetzner.com/tutorials/howto-hcloud-terrafo...

EDIT: For what it's worth, I have had good experiences with app servers hosted on Hetzner Cloud and managed Postgres provided by ElephantSQL (https://www.elephantsql.com/) for Germany-based apps.

replies(1): >>36812314 #
25. ◴[] No.36811969{5}[source]
26. fxtentacle ◴[] No.36812116[source]
I've been with them for a long time and my guesses would be:

1. Strict rules and strict customer verification. Crypto mining that wastes SSDs is not allowed. Portscans, mass emails, etc. are not allowed. They also don't offer GPUs to the general public because it has been abused in the past. You usually need to send in ID documents just to open an account. My guess is this allows them to avoid most bad actors and, thereby, waste less money on fraud.

2. Extremely long-term investments. They typically build their own hardware and then use it over 10 years. They have their own flea market where you can rent older server models for a steep discount. That means they will have a long time where the hardware is fully paid off and still generating revenue.

3. Great service. With a mid-sized company, I can call their technicians in the middle of the night. The fact that we could call them in case of a crisis has generated A LOT of good will. But I would be truly surprised if they didn't make a profit off those phone calls, as they charge roughly 4x the salary cost.

4. High-margin managed services. In addition to just the cheap servers, they also offer a managed service where they will do OS and security upgrades for you. It's roughly 2x the price of the server and it appears to be almost fully automated. I know some freelance web designers who will insist on using Hetzner Managed for deployment for their clients, because it is just so convenient. You effectively pass off all recurring maintenance for €300 a month and your client is happy to have an emergency phone number (see #3) in case the box goes down.

27. bigjoes ◴[] No.36812139{4}[source]
Could you please elaborate how and what you know about managed Kubernetes on Hetzner?

I am asking for this since a while and was told there is no way Hetzner would offer such a service. Certain Posts on Social Media have also never been answered with any kind of indication that they are actually working on it.

Please provide some Details on this.

replies(1): >>36812348 #
28. detourdog ◴[] No.36812142{4}[source]
I think multi-national energy sector should be working toward the goals without the regulations. The more prep done before the change the smoother the transition.
29. kinduff ◴[] No.36812172[source]
Same, tried a bunch before moving completely to Hetzner. I'm super happy with their service.
30. stevefan1999 ◴[] No.36812259{5}[source]
The competitor of DO, Vultr does this IIRC, yet it is not really cheaper
31. kristiandupont ◴[] No.36812314{4}[source]
Got it, thanks. I've used ElephantSQL as well and I've been happy with them.
32. dpeckett ◴[] No.36812348{5}[source]
They were in person recruiting at KubeCon EU this year and were advertising a good number of Kubernetes engineering roles. Definitely gave me the impression they were taking Kubernetes seriously but looking back a managed offering was just speculation on my part.

So huge grain of salt, you are totally right. It could be internal platform work only.

33. dpeckett ◴[] No.36812449{4}[source]
For anyone interested in Kubernetes on Hetzner, there's a really interesting CAPI provider being actively developed:

https://github.com/syself/cluster-api-provider-hetzner

34. KronisLV ◴[] No.36812572{3}[source]
> Simple, Hetzner mainly operates on Germany, the people are mostly Germans, and they automate the stuff to a point a small team could manage it well even if not remotely, so they have less cost on human resources.

I feel like there might be more to it, especially considering the situation with electricity prices in some places in EU recently.

I used (and still use) a Lithuanian platform called Time4VPS which was cheaper than Hetzner previously, yet had to increase their prices somewhat for that reason. Now only some of their plans are competitive with Hetzner, while Hetzner also provides some managed services as well.

Hetzner docs also went into some of the details regarding the pricing: https://docs.hetzner.com/robot/general/pricing/hetzner-prici...

And yet, I can't help but to wonder why they don't give in to the desire to maximize profit margins, like happened to say Scaleway (good platform, but as expensive as DigitalOcean).

35. victor106 ◴[] No.36812679{3}[source]
Can’t you use a third party IAM solution for this? Like Okta or keycloak?
replies(1): >>36846920 #
36. EspressoGPT ◴[] No.36812917[source]
Overstaffed, overinflated and inefficient Silicon Valley startup vs. organically-grown, well-adjusted, efficient German company.
replies(1): >>36813209 #
37. thejosh ◴[] No.36813024{3}[source]
Hetzner also do some crazy-cool stuff, especially around the 7950X3D, cooling, AM5 etc. (https://www.youtube.com/watch?v=V2P8mjWRqpk). They also do some amazing stuff with ARM (their cloud offering is really solid for this).
38. dangoodmanUT ◴[] No.36813077[source]
Hetzner has a record for going silent with issues FYI, just hit their reddit to see all the horror stories
39. wongarsu ◴[] No.36813209{3}[source]
Not to mention a German company that has price sensitivity in their DNA. Their first servers were just regular consumer tower PCs to drastically cut hardware costs. Now many years later it's a highly optimized mix of consumer, server and inhouse parts (e.g. they use their own racking system instead of 19", and the datacenters are built to make use of convection for a lot of the cooling). They also offer regular Dell servers for those that want them, but at 2x-4x the price of their homegrown boxes.
40. api ◴[] No.36813431{3}[source]
I've wondered how they can host this cheap in Germany given their very high electricity prices.

Maybe that's not actually the dominant cost, or they've optimized everything else so well they can just eat the electric bill.

41. ushakov ◴[] No.36813464{5}[source]
The secret is in the cooling system. They have individual cooling systems for each server. Less heat = longer sustained loads
replies(1): >>36824593 #
42. cutemonster ◴[] No.36814363{6}[source]
Where do DO get their servers and data centers from? ... Apparently they run on AWS, I'm surprised
replies(1): >>36834390 #
43. devjab ◴[] No.36815807{4}[source]
I think you might misunderstand me. The 2050 is a guesstimate and it's just my opinion on the matter. As far as planning ahead goes, you plan for 5-10 years when you try to figure out where to "iron" your enterprise IT. This is because that's how long your hardware will last if you go the route of renting rack space with your own hardware. I think we tend to plan for 8 years, with some space for "unintended" early failures on things like controllers after 4 years. So while you can contract big-cloud vendors for shorter, I think ours is on 3 year contracts right now, you still sort of do the business case for much longer. Maybe not every 3 years, but at least every 6 years.

You do the same on the other side of the table. Companies like Hetzner knows that EU cloud sollutions are likely to see growth, so it's only natural that they invest in the tech to put themselves in a prime position to jump on the opportunity. Selling a good product while you do so is the way I would do it personally, but you also have EU cloud initiatives backed by VC money going straight for the endgame.

44. abwizz ◴[] No.36824593{6}[source]
pardon my ingorance but i cannot quite see how cooling individual machines vs. the hole rack or row makes a difference in total heat production per machine
45. re-thc ◴[] No.36834390{7}[source]
> Apparently they run on AWS, I'm surprised

They don't run on AWS. Not sure what sort of rumors are running :(

> data centers from?

The major players e.g. Equinix, Coresite, etc. Varies per location. Even AWS don't build most of their data centers.

46. mffap ◴[] No.36846920{4}[source]
zitadel supports service users with rbac. maybe give it a look/try: https://github.com/zitadel/zitadel
47. hardwaresofton ◴[] No.36857655[source]
Hey would you be into trying a manage service platform I'm building for Hetzner? It's called Nimbus[0].

I'd love some feedback, specifically:

- Which services do you most want to use/have managed

- What databases do you find yourself using the most

- Concerning caches, do you use memcached or mostly Redis?

[0]: https://nimbusws.com

48. hardwaresofton ◴[] No.36857753[source]
Hey I'm building a managed service platform (not quite an app store!) on top of Hetzner -- would you be interested in trying it out?

Contact is in my profile but I'd love to have some more people kick the tires and tell me what they want built the most.