Hard drives on backorder for two years as AI data centers trigger HDD shortage

It's rough out there and has become increasingly difficult to maintain our pace of storage deployment.

Further - and most concerning - is the pollution of the supply chain with refurbished/recertified stock being sold and marketed as "new".

One example:

https://kozubik.com/items/MaestroTechnology/

I strongly advise buyers to stick with trusted suppliers, avoid Amazon/ebay channels, and carefully vet your incoming stock with SMART tools to ensure you receive what you think you are ... especially for SSD parts.

Question for all of you more knowledgeble than I: can SMART data be tampered with? When I get, say, a refurbished Mac from Apple, I'm trusting Apple won't stoop to that. But a SSD vendor I've never heard of?

Yes, it can be tampered with. Drives can even lie about the amount of storage they support. I once bought a 1TB pen drive that was only 32MB for $10. (Yes, I knew it was a scam beforehand.)

DO NOT assume SMART is reliable. You can wipe SMART stats or write any values you want.

You have to actually examine the real bits on the drive. Resellers don't want to take the time to actually zero a drive, they usually just nuke the partition table.

You also need to physically examine the drive. Corroded fingerprints on the PCB, wear on the port contacts, scratches from mounting rails, etc.

That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

Physical examination is the only reliable method.

> That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.

This sounds like it could be a big problem for Backblaze customers, and consequently for Backblaze.

Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?

Backblaze customers also need to know, but I would give Backblaze the first shot at figuring out how to notify, whom, of what.

> drive I bought on Amazon was actually a used Backblaze drive

Assuming this is true, I find it weird/surprising that Backblaze doesn't at least zero their drives before disposing of them? I have to do that at my work, and at least by policy I could lose my job if I skipped doing it.

Yes. There are vendor-specific utilities that have escaped into the wild that allow bad actors to reset various SMART counters, etc.

A lot of abuse came to light during the launch and initial mining of the (ridiculous) Chiacoin[1] during which Chia miners would burn through SSDs to within a hair of their usable life, reset their SMART stats, and sell them as new on Amazon or ebay.

As can be seen in my above comment, larger distributors like "Maestro Technologies" have their stock polluted with parts like this and I find it very unlikely that they are not aware of the status of these parts they are selling as new.

[1] https://en.wikipedia.org/wiki/Chia_Network

But you don’t work at backblaze :)

Nearly any product you can buy from Amazon, even when it says shipped from Amazon, is suspect.

I wouldn't shop there at all. It's a literal scam market. Allegedly.

I find it more weird that they don't use encrypted storage, then you don't nee to bother with zeroing drives. You only need to 'forget' the key.

Backblaze erasure-codes customer data across 17 (I think) servers, so customer data is probably not accessible. Yes, it would be better if they zeroed the drive, but Google says that will take 14-30 hours for a 10TB drive.

For drives that implement an internal encryption key, it's faster (instantaneous) to reset the encryption key. It won't give you a zeroed drive, but one filled with garbage.

The earlier description is ambiguous (i.e., is it data of or about customers, and is that data cleartext), but it seems they believe they have a drive from Backblaze with a lot of cleartext files on it, and something involving customers.

> It contained terabytes of customer data, and a shit ton of cleartext files.

In many erasure coding systems, the first X sets of code are simply cleartext chunks.

This is also more efficient in the happy path since then no computation is needed to decode the data. It can be DMA'd straight from the drive to the network adapter with super low CPU utilisation even for Gbps of network traffic.