←back to thread

SSDs have become fast, except in the cloud

(databasearchitects.blogspot.com)
589 points greghn | 2 comments | 20 Feb 24 16:59 UTC | HN request time: 0.736s | source
Show context
pclmulqdq ◴[20 Feb 24 17:22 UTC] No.39443994[source]▶
This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions.

SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD.

replies(30): >>39444009 #>>39444024 #>>39444028 #>>39444046 #>>39444062 #>>39444085 #>>39444096 #>>39444099 #>>39444120 #>>39444138 #>>39444328 #>>39444374 #>>39444396 #>>39444429 #>>39444655 #>>39444952 #>>39445035 #>>39445917 #>>39446161 #>>39446248 #>>39447169 #>>39447467 #>>39449080 #>>39449287 #>>39449377 #>>39449994 #>>39450169 #>>39450172 #>>39451330 #>>39466088 #
jsnell ◴[20 Feb 24 17:31 UTC] No.39444096[source]▶
According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions.

Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

replies(3): >>39444161 #>>39444353 #>>39448728 #
1. crazygringo ◴[20 Feb 24 17:50 UTC] No.39444353[source]▶
> It's about SSDs being two orders of magnitude slower than datacenter networks.

Could that have to do with every operation requiring a round trip, rather than being able to queue up operations in a buffer to saturate throughput?

It seems plausible if the interface protocol was built for a device it assumed was physically local and so waited for confirmation after each operation before performing the next.

In this case it's not so much the throughput rate that matters, but the latency -- which can also be heavily affected by buffering of other network traffic.

replies(1): >>39444467 #
2. Nextgrid ◴[20 Feb 24 17:58 UTC] No.39444467[source]▶
Underlying protocol limitations wouldn't be an issue - the cloud provider's implementation can work around that. They're unlikely to be sending sequential SCSI/NVMe commands over the wire - instead, the hypervisor pretends to be the NVME device, but then converts to some internal protocol (that's less chatty and can coalesce requests without waiting on individual ACKs) before sending that to the storage server.

The problem is that ultimately your application often requires the outcome of a given IO operation to decide which operation to perform next - let's say when it comes to a database, it should first read the index (and wait for that to complete) before it knows the on-disk location of the actual row data which it needs to be able to issue the next IO operation.

In this case, there's no other solution than to move that application closer to the data itself. Instead of the networked storage node being a dumb blob storage returning bytes, the networked "storage" node is your database itself, returning query results. I believe that's what RDS Aurora does for example, every storage node can itself understand query predicates.