←back to thread

WebSockets cost us $1M on our AWS bill

(www.recall.ai)
257 points tosh | 6 comments | 06 Nov 24 18:50 UTC | HN request time: 1.117s | source | bottom
Show context
turtlebits ◴[06 Nov 24 19:58 UTC] No.42068395[source]▶
Is this really an AWS issue? Sounds like you were just burning CPU cycles, which is not AWS related. WebSockets makes it sound like it was a data transfer or API gateway cost.
replies(2): >>42068522 #>>42068890 #
VWWHFSfQ ◴[06 Nov 24 20:06 UTC] No.42068522[source]▶
> Is this really an AWS issue?

I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.

So yes, it's an AWS issue.

replies(1): >>42068676 #
JackSlateur ◴[06 Nov 24 20:17 UTC] No.42068676[source]▶

  You can rent real 44-core Xeon servers for like, $250/month.
Where, for instance ?
replies(3): >>42068729 #>>42068739 #>>42068788 #
VWWHFSfQ ◴[06 Nov 24 20:25 UTC] No.42068788[source]▶
There are many colos that offer dedicated server rental/hosting. You can just google for colos in the region you're looking for. I found this one

https://www.colocrossing.com/server/dedicated-servers/

replies(1): >>42068853 #
petcat ◴[06 Nov 24 20:30 UTC] No.42068853[source]▶
I don't know anything about Colo Crossing (are they a reseller?) but I would bet their $60 per month 4-core Intel Xeons would outperform a $1,000 per month "compute optimized" EC2 server.
replies(2): >>42068909 #>>42069533 #
phonon ◴[06 Nov 24 21:16 UTC] No.42069533[source]▶
For $1000 per month you can get a c8g.12xlarge (assuming you use some kind of savings plan).[0] That's 48 cores, 96 GB of RAM and 22.5+ Gbps networking. Of course you still need to pay for storage, egress etc., but you seem to be exaggerating a bit....they do offer a 44 core Broadwell/128 GB RAM option for $229 per month, so AWS is more like a 4x markup[1]....the C8g would likely be much faster at single threaded tasks though[2][3]

[0]https://instances.vantage.sh/aws/ec2/c8g.12xlarge?region=us-... [1]https://portal.colocrossing.com/register/order/service/480 [2]https://browser.geekbench.com/v6/cpu/8305329 [3]https://browser.geekbench.com/processors/intel-xeon-e5-2699-...

replies(1): >>42070292 #
petcat ◴[06 Nov 24 22:12 UTC] No.42070292[source]▶
> That's 48 cores

That's not dedicated 48 cores, it's 48 "vCPUs". There are probably 1,000 other EC2 instances running on those cores stealing all the CPU cycles. You might get 4 cores of actual compute throughput. Which is what I was saying

replies(1): >>42070456 #
phonon ◴[06 Nov 24 22:24 UTC] No.42070456[source]▶
That's not how it works, sorry. (Unless you use burstable instances, like T4g) You can run them at 100% as long as you like, and it has the same performance (minus a small virtualization overhead).
replies(1): >>42070772 #
1. petcat ◴[06 Nov 24 22:48 UTC] No.42070772[source]▶
Are you telling me that my virtualized EC2 server is the only thing running on the physical hardware/CPU? There are no other virtualized EC2 servers sharing time on that hardware/CPU?
replies(1): >>42071053 #
2. phonon ◴[06 Nov 24 23:16 UTC] No.42071053[source]▶
If you are talking about regular EC2 (not T series, or Lambda, or Fargate etc.) you get the same performance (within say 5%) of the underlying hardware. If you're using a core, it's not shared with another user. The pricing validates this...the "metal" version of a server on AWS is the same price as the full regular EC2 version.

In fact, you can even get a small discount with the -flex series, if you're willing to compromise slightly. (Small discount for 100% of performance 95% of the time).

replies(1): >>42071531 #
3. petcat ◴[07 Nov 24 00:00 UTC] No.42071531[source]▶
This seems pretty wild to me. Are you saying that I can submit instructions to the CPU and they will not be interleaved and the registers will not be swapped-out with instructions from other EC2 virtual server applications running on the same physical machine?
replies(3): >>42072496 #>>42072775 #>>42072840 #
4. phonon ◴[07 Nov 24 02:03 UTC] No.42072496{3}[source]▶
Welcome to the wonderful world of multi-core CPUs...
5. nostrebored ◴[07 Nov 24 02:41 UTC] No.42072775{3}[source]▶
Yes — you can validate this by benchmarking things like l1 cache
6. doctorpangloss ◴[07 Nov 24 02:49 UTC] No.42072840{3}[source]▶
Only the t instances and other VM types that have burst billing are overbooked in the sense that you are describing.