Most active commenters
  • dilyevsky(4)
  • dijit(4)

←back to thread

WebSockets cost us $1M on our AWS bill

(www.recall.ai)
261 points tosh | 15 comments | 06 Nov 24 18:50 UTC | HN request time: 0.726s | source | bottom
Show context
turtlebits ◴[06 Nov 24 19:58 UTC] No.42068395[source]▶
Is this really an AWS issue? Sounds like you were just burning CPU cycles, which is not AWS related. WebSockets makes it sound like it was a data transfer or API gateway cost.
replies(2): >>42068522 #>>42068890 #
VWWHFSfQ ◴[06 Nov 24 20:06 UTC] No.42068522[source]▶
> Is this really an AWS issue?

I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.

So yes, it's an AWS issue.

replies(1): >>42068676 #
JackSlateur ◴[06 Nov 24 20:17 UTC] No.42068676[source]▶

  You can rent real 44-core Xeon servers for like, $250/month.
Where, for instance ?
replies(3): >>42068729 #>>42068739 #>>42068788 #
1. Faaak ◴[06 Nov 24 20:21 UTC] No.42068729[source]▶
Hetzner for example. An EPYC 48c (96t) goes for 230 euros
replies(1): >>42068782 #
2. dilyevsky ◴[06 Nov 24 20:24 UTC] No.42068782[source]▶
Hetzner network is complete dog. They also sell you machines that are long should be EOL’ed. No serious business should be using them
replies(3): >>42068965 #>>42069178 #>>42069210 #
3. dijit ◴[06 Nov 24 20:37 UTC] No.42068965[source]▶
What cpu do you think your workload is using on AWS?

GCP exposes their cpu models, and they have some Haswell and Broadwell lithographies in service.

Thats a 10+ year old part, for those paying attention.

replies(2): >>42069283 #>>42069684 #
4. speedgoose ◴[06 Nov 24 20:52 UTC] No.42069178[source]▶
I know serious businesses using Hetzner for their critical workloads. I wouldn’t unless money is tight, but it is possible. I use them for my non critical stuff, it costs so much less.
5. blibble ◴[06 Nov 24 20:54 UTC] No.42069210[source]▶
I just cat'ed /proc/cpuinfo on my Hetzner and AWS machines

AWS: E5-2680 v4 (2016)

Hetzner: Ryzen 5 (2019)

replies(1): >>42071461 #
6. dilyevsky ◴[06 Nov 24 20:58 UTC] No.42069283{3}[source]▶
Most of GCP and some AWS instances will migrate to another node when it’s faulty. Also disk is virtual. None of this applies to baremetal hetzner
replies(1): >>42069319 #
7. dijit ◴[06 Nov 24 21:00 UTC] No.42069319{4}[source]▶
Why is that relevant to what I said?
replies(1): >>42069607 #
8. dilyevsky ◴[06 Nov 24 21:21 UTC] No.42069607{5}[source]▶
Only relevant if you care about reliability
replies(1): >>42069669 #
9. dijit ◴[06 Nov 24 21:25 UTC] No.42069669{6}[source]▶
AWS was working “fine” for about 10 years without live migration, and I’ve had several individual machines running without a reboot or outage for quite literally half a decade. Enough to hit bugs like this: https://support.hpe.com/hpesc/public/docDisplay?docId=a00092...

Anyway, depending on individual nodes to always be up for reliability is incredibly foolhardy. Things can happen, cloud isn't magic, I’ve had instances become unrecoverable. Though it is rare.

So, I still don’t understand the point, that was not exactly relevant to what I said.

10. tsimionescu ◴[06 Nov 24 21:26 UTC] No.42069684{3}[source]▶
I think they meant that Hetzner is offering specific machines they know to be faulty and should have EOLd to customers, not that they use deprecated CPUs.
replies(1): >>42069729 #
11. dijit ◴[06 Nov 24 21:30 UTC] No.42069729{4}[source]▶
Thats scary if true, any sources? My google-fu is failing me. :/
replies(1): >>42070304 #
12. akvadrako ◴[06 Nov 24 22:13 UTC] No.42070304{5}[source]▶
It's not scary, it's part of the value proposition.

I used to work for a company that rented lots of hetzner boxes. Consumer grade hardware with frequent disk failures was just what we excepted for saving a buck.

13. dilyevsky ◴[06 Nov 24 23:53 UTC] No.42071461{3}[source]▶
Now do hard drives
replies(2): >>42072132 #>>42072929 #
14. blibble ◴[07 Nov 24 01:08 UTC] No.42072132{4}[source]▶
the hetzner one is a dedicated pcie 4.0 nvme device and wrote at 2.3GB/s (O_DIRECT)

the AWS one is some emulated block device, no idea what it is, other than it's 20x slower

15. luuurker ◴[07 Nov 24 03:05 UTC] No.42072929{4}[source]▶
You keep moving the goal posts with these replies.

Hetzner isn't the best provider in the world, but it's also not as bad as you say they are. They're not just renting old servers.