←back to thread

Hard drives on backorder for two years as AI data centers trigger HDD shortage

(www.tomshardware.com)
138 points pabs3 | 8 comments | 12 Nov 25 05:36 UTC | HN request time: 1.203s | source | bottom
Show context
rsync ◴[12 Nov 25 16:41 UTC] No.45902291[source]▶
It's rough out there and has become increasingly difficult to maintain our pace of storage deployment.

Further - and most concerning - is the pollution of the supply chain with refurbished/recertified stock being sold and marketed as "new".

One example:

https://kozubik.com/items/MaestroTechnology/

I strongly advise buyers to stick with trusted suppliers, avoid Amazon/ebay channels, and carefully vet your incoming stock with SMART tools to ensure you receive what you think you are ... especially for SSD parts.

replies(3): >>45903958 #>>45904782 #>>45905884 #
1. estimator7292 ◴[12 Nov 25 19:17 UTC] No.45904782[source]▶
DO NOT assume SMART is reliable. You can wipe SMART stats or write any values you want.

You have to actually examine the real bits on the drive. Resellers don't want to take the time to actually zero a drive, they usually just nuke the partition table.

You also need to physically examine the drive. Corroded fingerprints on the PCB, wear on the port contacts, scratches from mounting rails, etc.

That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

Physical examination is the only reliable method.

replies(2): >>45905070 #>>45905096 #
2. neilv ◴[12 Nov 25 19:31 UTC] No.45905070[source]▶
> That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

This sounds like it could be a big problem for Backblaze customers, and consequently for Backblaze.

Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?

Backblaze customers also need to know, but I would give Backblaze the first shot at figuring out how to notify, whom, of what.

replies(1): >>45908285 #
3. SoftTalker ◴[12 Nov 25 19:33 UTC] No.45905096[source]▶
> drive I bought on Amazon was actually a used Backblaze drive

Assuming this is true, I find it weird/surprising that Backblaze doesn't at least zero their drives before disposing of them? I have to do that at my work, and at least by policy I could lose my job if I skipped doing it.

replies(2): >>45905560 #>>45908284 #
4. loloquwowndueo ◴[12 Nov 25 19:59 UTC] No.45905560[source]▶
But you don’t work at backblaze :)
5. sigio ◴[12 Nov 25 23:18 UTC] No.45908284[source]▶
I find it more weird that they don't use encrypted storage, then you don't nee to bother with zeroing drives. You only need to 'forget' the key.
6. prirun ◴[12 Nov 25 23:18 UTC] No.45908285[source]▶
Backblaze erasure-codes customer data across 17 (I think) servers, so customer data is probably not accessible. Yes, it would be better if they zeroed the drive, but Google says that will take 14-30 hours for a 10TB drive.

For drives that implement an internal encryption key, it's faster (instantaneous) to reset the encryption key. It won't give you a zeroed drive, but one filled with garbage.

replies(2): >>45908533 #>>45908586 #
7. neilv ◴[12 Nov 25 23:44 UTC] No.45908533{3}[source]▶
The earlier description is ambiguous (i.e., is it data of or about customers, and is that data cleartext), but it seems they believe they have a drive from Backblaze with a lot of cleartext files on it, and something involving customers.

> It contained terabytes of customer data, and a shit ton of cleartext files.

8. londons_explore ◴[12 Nov 25 23:51 UTC] No.45908586{3}[source]▶
In many erasure coding systems, the first X sets of code are simply cleartext chunks.

This is also more efficient in the happy path since then no computation is needed to decode the data. It can be DMA'd straight from the drive to the network adapter with super low CPU utilisation even for Gbps of network traffic.