←back to thread

WebSockets cost us $1M on our AWS bill

(www.recall.ai)
257 points tosh | 8 comments | 06 Nov 24 18:50 UTC | HN request time: 0.001s | source | bottom
Show context
turtlebits ◴[06 Nov 24 19:58 UTC] No.42068395[source]▶
Is this really an AWS issue? Sounds like you were just burning CPU cycles, which is not AWS related. WebSockets makes it sound like it was a data transfer or API gateway cost.
replies(2): >>42068522 #>>42068890 #
VWWHFSfQ ◴[06 Nov 24 20:06 UTC] No.42068522[source]▶
> Is this really an AWS issue?

I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.

So yes, it's an AWS issue.

replies(1): >>42068676 #
JackSlateur ◴[06 Nov 24 20:17 UTC] No.42068676[source]▶

  You can rent real 44-core Xeon servers for like, $250/month.
Where, for instance ?
replies(3): >>42068729 #>>42068739 #>>42068788 #
Faaak ◴[06 Nov 24 20:21 UTC] No.42068729[source]▶
Hetzner for example. An EPYC 48c (96t) goes for 230 euros
replies(1): >>42068782 #
dilyevsky ◴[06 Nov 24 20:24 UTC] No.42068782[source]▶
Hetzner network is complete dog. They also sell you machines that are long should be EOL’ed. No serious business should be using them
replies(3): >>42068965 #>>42069178 #>>42069210 #
1. dijit ◴[06 Nov 24 20:37 UTC] No.42068965[source]▶
What cpu do you think your workload is using on AWS?

GCP exposes their cpu models, and they have some Haswell and Broadwell lithographies in service.

Thats a 10+ year old part, for those paying attention.

replies(2): >>42069283 #>>42069684 #
2. dilyevsky ◴[06 Nov 24 20:58 UTC] No.42069283[source]▶
Most of GCP and some AWS instances will migrate to another node when it’s faulty. Also disk is virtual. None of this applies to baremetal hetzner
replies(1): >>42069319 #
3. dijit ◴[06 Nov 24 21:00 UTC] No.42069319[source]▶
Why is that relevant to what I said?
replies(1): >>42069607 #
4. dilyevsky ◴[06 Nov 24 21:21 UTC] No.42069607{3}[source]▶
Only relevant if you care about reliability
replies(1): >>42069669 #
5. dijit ◴[06 Nov 24 21:25 UTC] No.42069669{4}[source]▶
AWS was working “fine” for about 10 years without live migration, and I’ve had several individual machines running without a reboot or outage for quite literally half a decade. Enough to hit bugs like this: https://support.hpe.com/hpesc/public/docDisplay?docId=a00092...

Anyway, depending on individual nodes to always be up for reliability is incredibly foolhardy. Things can happen, cloud isn't magic, I’ve had instances become unrecoverable. Though it is rare.

So, I still don’t understand the point, that was not exactly relevant to what I said.

6. tsimionescu ◴[06 Nov 24 21:26 UTC] No.42069684[source]▶
I think they meant that Hetzner is offering specific machines they know to be faulty and should have EOLd to customers, not that they use deprecated CPUs.
replies(1): >>42069729 #
7. dijit ◴[06 Nov 24 21:30 UTC] No.42069729[source]▶
Thats scary if true, any sources? My google-fu is failing me. :/
replies(1): >>42070304 #
8. akvadrako ◴[06 Nov 24 22:13 UTC] No.42070304{3}[source]▶
It's not scary, it's part of the value proposition.

I used to work for a company that rented lots of hetzner boxes. Consumer grade hardware with frequent disk failures was just what we excepted for saving a buck.