Upgrading an M4 Pro Mac mini's storage for half the price

About a decade and a half ago, Apple paid half a billion dollars to acquire the patents of a company making enterprise SSD controllers.

> Anobit appears to be applying a lot of signal processing techniques in addition to ECC to address the issue of NAND reliability and data retention. In its patents there are mentions of periodically refreshing cells whose voltages may have drifted, exploiting some of the behaviors of adjacent cells and generally trying to deal with the things that happen to NAND once it's been worn considerably.

Through all of these efforts, Anobit is promising significant improvements in NAND longevity and reliability.

https://www.anandtech.com/show/5258/apple-acquires-anobit-br...

Every flash controller does this. Modern NAND is just math on a stick. Lots and lots of math.

Still sucks that you can’t use standard parts.

If they're really interested with data integrity they should add checksums to APFS.

If you don't have RAID you can't rebuild corrupted data, but at least you know there's a problem and perhaps restore from Time Machine.

For metadata, you may have multiple copies, so can use a known-good one (this is how ZFS works: some things have multiple copies 'inherently' because they're so important).

Edit:

> Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18]

* https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...

https://forums.macrumors.com/threads/upgrading-2013-2015-mac...

There is no reason to speculate as the reason is know (as stated by Jeff Bonwick, one of the co-inventors of ZFS):

>> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms.

> I cannot disclose details, but that is the essence of it.

* https://archive.is/http://mail.opensolaris.org/pipermail/zfs...

* https://web.archive.org/web/*/http://mail.opensolaris.org/pi...

Same thing with DDR5: the electrical layer is a beast, it's a reason enough to require its own controller.

> Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18]

* https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...

Or you can spend half a billion dollars to solve the issue in hardware.

As one of the creators of ZFS wrote when APFS was announced:

> Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both NAND flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The Apple engineers contend that Apple devices basically don't return bogus data.

https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana...

APFS keeps redundant copies and checksums for metadata, but doesn't constantly checksum files looking for changes any more than NTFS does.

On the CPU's PCIe bus. NVMe drives are PCIe devices, designed specifically to facilitate such interfacing.

Edit: Pardon, misread the actual statement you responded to. Of course one shouldn't hook NAND directly to the CPU. I'll leave my response for whatever value the info has.

Handling file integrity at the hardware level is a big step up.

And hope that your hardware/firmware doesn't ever get bugs.

Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth.

ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down":

* https://www.youtube.com/watch?v=fE2KDzZaxvE

And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all.

Why not come up with a solution that covers external storage too, instead of spending all that money and relying on external solutions? I just don't understand why they couldn't have optional checksums in APFS.

For instance, try to find a processor aimed at mobile devices that doesn't handle video decoding in dedicated hardware instead of running it on a CPU core.

By using external storage, instead of paying $10k more for more storage, you are directly harming Apple’s margins and the CEO’s bonus which is not ok /s

ReFS exists, so Microsoft knew they needed to do something, but they have utterly failed to protect the vast majority of users.

We already see the demand for this in the latest NVMe protocol spec that allows the host to give placement hints. But this is a half-measure that suggests what systems really want, which is not to vaguely influence the device but instead to tell it exactly what to do.

I know if my previous job at a large hard drive manufacturer we had special Apple drives that ran different parts and firmware than the regular PC drives. Their specs and tolerances where much different than the PC market at a whole.

Odds are very good that totally different people work on the architecture of AFS and SoC design.

Don't get me wrong, there's no reason Microsoft can't transition to another filesystem (like offering ReFS outside of Server or whatever Windows variants support it currently), but I don't understand why a company would transition to a new filesystem in 2016 and not include a data checksums option. Hell, ReFS predates APFS, and I think it even has optional data checksums.

Fixed links for the message and reply:

https://web.archive.org/web/20091028/http://mail.opensolaris...

https://web.archive.org/web/20091028/http://mail.opensolaris...

Handling file integrity in hardware is a big step up.

Is there any evidence that Apple actually has better hardware data integrity than anyone else, though? They make claims in the article linked a few posts back, but AFAIK SSDs in general make use of error correcting codes, not just Apple's SSDs.

That article also points out how even multi-million dollar arrays are known to return bad data, and previous Apple SSD devices have been known to do the same.

I agree that the state of default filesystems is bad, but I'm not convinced that Apple's hardware solution is anything more than them saying, "Trust me, bro."

https://forums.guru3d.com/threads/testing-instaling-windows-...

1. There is hardware support for (e.g.) SHA in ARM:

* https://developer.arm.com/documentation/ddi0514/g/introducti...

But given Apple designs their own CPUs they could add extensions for anything they need. Or use a simpler algorithm, like Fletcher (which ZFS uses):

* https://en.wikipedia.org/wiki/Fletcher%27s_checksum

2. It does not have to be enabled by default for every device. The main problem is the lack of it even as an option.

I wouldn't necessarily use ZFS checksums on a laptop, but ZFS has them for when I use it on a not-laptop.

We’re talking “watch the mouse crawl across the machine” crushed. Completely useless. Life returned to normal when I uninstalled it.

Also, I’ve heard anecdotes that ZFS and USB do not get along.

I’ve also heard contrary experiences. Some folks, somewhere, may be having success with ZFS on external drives on an iMac.

I’m just not one of them.

Indeed. They added an entire enterprise grade SSD controller.

> In its patents there are mentions of periodically refreshing cells whose voltages may have drifted, exploiting some of the behaviors of adjacent cells and generally trying to deal with the things that happen to NAND once it's been worn considerably.

https://git.eeqj.de/sneak/attrsum

I use zfs where I can (it has content checksums) but it sucks bad on macOS, so I wrote attrsum. It keeps the file content checksum in an xattr (which APFS (and ext3/4) supports).

I use it to protect my photo library on a huge external SSD formatted with APFS (encrypted, natch) because I need to mount it on a mac laptop for Lightroom.

https://eclecticlight.co/dintch/