WebSockets cost us $1M on our AWS bill

Is this really an AWS issue? Sounds like you were just burning CPU cycles, which is not AWS related. WebSockets makes it sound like it was a data transfer or API gateway cost.

> Is this really an AWS issue?

I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.

So yes, it's an AWS issue.

  You can rent real 44-core Xeon servers for like, $250/month.

Where, for instance ?

Hetzner for example. An EPYC 48c (96t) goes for 230 euros

Hetzner: https://www.hetzner.com/dedicated-rootserver/#cores_threads_...

Hetzner network is complete dog. They also sell you machines that are long should be EOL’ed. No serious business should be using them

There are many colos that offer dedicated server rental/hosting. You can just google for colos in the region you're looking for. I found this one

https://www.colocrossing.com/server/dedicated-servers/

I don't know anything about Colo Crossing (are they a reseller?) but I would bet their $60 per month 4-core Intel Xeons would outperform a $1,000 per month "compute optimized" EC2 server.

Neither the title nor the article are painting it as an AWS issue, but as a websocket issue, because the protocol implicitly requires all transferred data to be copied multiple times.

What benchmark would you like to use?

What cpu do you think your workload is using on AWS?

GCP exposes their cpu models, and they have some Haswell and Broadwell lithographies in service.

Thats a 10+ year old part, for those paying attention.

This blog is about doing video processing on the CPU, so something akin to that.

I know serious businesses using Hetzner for their critical workloads. I wouldn’t unless money is tight, but it is possible. I use them for my non critical stuff, it costs so much less.

I just cat'ed /proc/cpuinfo on my Hetzner and AWS machines

AWS: E5-2680 v4 (2016)

Hetzner: Ryzen 5 (2019)

Most of GCP and some AWS instances will migrate to another node when it’s faulty. Also disk is virtual. None of this applies to baremetal hetzner

Why is that relevant to what I said?

For $1000 per month you can get a c8g.12xlarge (assuming you use some kind of savings plan).[0] That's 48 cores, 96 GB of RAM and 22.5+ Gbps networking. Of course you still need to pay for storage, egress etc., but you seem to be exaggerating a bit....they do offer a 44 core Broadwell/128 GB RAM option for $229 per month, so AWS is more like a 4x markup[1]....the C8g would likely be much faster at single threaded tasks though[2][3]

[0]https://instances.vantage.sh/aws/ec2/c8g.12xlarge?region=us-... [1]https://portal.colocrossing.com/register/order/service/480 [2]https://browser.geekbench.com/v6/cpu/8305329 [3]https://browser.geekbench.com/processors/intel-xeon-e5-2699-...

Only relevant if you care about reliability

AWS was working “fine” for about 10 years without live migration, and I’ve had several individual machines running without a reboot or outage for quite literally half a decade. Enough to hit bugs like this: https://support.hpe.com/hpesc/public/docDisplay?docId=a00092...

Anyway, depending on individual nodes to always be up for reliability is incredibly foolhardy. Things can happen, cloud isn't magic, I’ve had instances become unrecoverable. Though it is rare.

So, I still don’t understand the point, that was not exactly relevant to what I said.

I think they meant that Hetzner is offering specific machines they know to be faulty and should have EOLd to customers, not that they use deprecated CPUs.

Thats scary if true, any sources? My google-fu is failing me. :/

If you call out your vendor, the issue usually lies with some specific issue with them or their service. The title obviously states AWS.

If I said that "childbirth cost us 5000 on our <hospital name> bill", you assume the issue is with the hospital.

> That's 48 cores

That's not dedicated 48 cores, it's 48 "vCPUs". There are probably 1,000 other EC2 instances running on those cores stealing all the CPU cycles. You might get 4 cores of actual compute throughput. Which is what I was saying

It's not scary, it's part of the value proposition.

I used to work for a company that rented lots of hetzner boxes. Consumer grade hardware with frequent disk failures was just what we excepted for saving a buck.

I disagree. Like @turtlebits, I was waiting for the part of the story where websocket connections between their AWS resources somehow got billed at Amazon's internet data egress rates.

That's not how it works, sorry. (Unless you use burstable instances, like T4g) You can run them at 100% as long as you like, and it has the same performance (minus a small virtualization overhead).

I didn't know this - why is this the case?

Are you telling me that my virtualized EC2 server is the only thing running on the physical hardware/CPU? There are no other virtualized EC2 servers sharing time on that hardware/CPU?

If you are talking about regular EC2 (not T series, or Lambda, or Fargate etc.) you get the same performance (within say 5%) of the underlying hardware. If you're using a core, it's not shared with another user. The pricing validates this...the "metal" version of a server on AWS is the same price as the full regular EC2 version.

In fact, you can even get a small discount with the -flex series, if you're willing to compromise slightly. (Small discount for 100% of performance 95% of the time).

Only for people that just read headlines and make technical decisions based on them. Are we catering to them now? The title is factual and straightforward.

Now do hard drives

This seems pretty wild to me. Are you saying that I can submit instructions to the CPU and they will not be interleaved and the registers will not be swapped-out with instructions from other EC2 virtual server applications running on the same physical machine?

the hetzner one is a dedicated pcie 4.0 nvme device and wrote at 2.3GB/s (O_DIRECT)

the AWS one is some emulated block device, no idea what it is, other than it's 20x slower

Welcome to the wonderful world of multi-core CPUs...

Yes — you can validate this by benchmarking things like l1 cache

Only the t instances and other VM types that have burst billing are overbooked in the sense that you are describing.

You keep moving the goal posts with these replies.

Hetzner isn't the best provider in the world, but it's also not as bad as you say they are. They're not just renting old servers.

Sorry, I have no idea if this is true. I was just pointing out what the GP was trying to claim.

And also highlights a meaningful irrelevance.

The idea that clearer titles are just babying some class of people is perverse.

Titles are the foremost means of deciding what to read, for anyone of any sophistication. Clearer titles benefit everyone.

The subject matter is meaningful to more than AWS users, but non-AWS users are going to be less likely to read it based on the title.

Wouldn't c8g.12xlarge with 500g storage (only EBS is possible), plus 1gbps from/to the internet is 5,700 USD per month, that's some discount you have.

If I try to match the actual machine. 16G ram. A rough estimate is that their Xeon E3-1240 would be ~2 AWS vCPU. So an r6g.large is the instance that would roughly match this one. Add 500G disk + 1 Gbps to/from the internet and ... monthly cost 3,700 USD.

Without any disk and without any data transfer (which would be unusable) it's still ~80USD. Maybe you could create a bootable image that calculates primes.

These are still not the same thing, I get it, but ... it's safe to say you cannot get anything remotely comparable on AWS. You can only get a different thing for way more money.

(made estimates on https://calculator.aws/ )

I checked here: https://www.hetzner.com/managed-server/

I see "AMD EPYC 7502P 32-Core" for 236 EUR per month. Can you tell me where you see 48c/96t?

EDIT

I found it! Unbelievable that it is so cheap.

https://www.hetzner.com/dedicated-rootserver/#cores_threads_...

What do you mean by "1gbps from/to the internet"?

125 MB per second × 60 seconds per minute × 60 minutes per hour × 24 hours per day x 30 days = 324 TB?

If you want 1 Gbps unmetered colo pricing, AWS is not competitive. So set up your video streaming service elsewhere :-)

https://portal.colocrossing.com/register/order/service/480 offers unmetered for $2,500 additional per month, for the record.

If you have high bandwidth needs on AWS you can use AWS Lightsail, which has some discounted transfer rates.

Even just the compute, without even disk, is barely competitive.

I'm not sure I understand your point anymore.