There is no 'modern' ZFS-like fs in Linux nowadays.
zfs is out of tree leaving it as an unviable option for many people. This news means that bcachefs is going to be in a very weird state in-kernel, which leaves only btrfs as the only other in-tree ‘modern’ filesystem.
This news about bcachefs has ramifications about the state of ‘modern’ FSes in Linux, and I’d say this news about the btrfs maintainer taking a step back is related to this.
1. The dm layer gives you cow/snapshots for any filesystem you want already and has for more than a decade. Some implementations actually use it for clever trickery like updates, even. Anyone who has software requirements in this space (as distinct from "wants to yell on the internet about it") is very well served.
2. Compression seems silly in the modern world. Virtually everything is already compressed. To first approximation, every byte in persistent storage anywhere in the world is in a lossy media format. And the ones that aren't are in some other cooked format. The only workloads where you see significant use of losslessly-compressible data are in situations (databases) where you have app-managed storage performance (and who see little value from filesystem choice) or ones (software building, data science, ML training) where there's lots of ephemeral intermediate files being produced. And again those are usages where fancy filesystems are poorly deployed, you're going to throw it all away within hours to days anyway.
Filesystems are a solved problem. If ZFS disappeared from the world today... really who would even care? Only those of us still around trying to shout on the internet.
Yeah nah, have you tried processing terabytes of data every day and storing them? It gets better now with DDR5 but bit flips do actually happen.
O_o
Apparently I've been living under a rock, can you please show us a link about this? I was just recently (casually) looking into bolting ZFS/BTRFS-like partial snapshot features to simulate my own atomic distro where I am able to freely roll back if an update goes bad. Think Linux's Timeshift with something little extra.
https://docs.kernel.org/admin-guide/device-mapper/snapshot.h...
As for the snapshots, things like LVM snapshots are pretty coarse, especially for someone like me where I run dm-crypt on top of LVM
I’d say zfs would be pretty well missed with its data integrity features. I’ve heard that btrfs is worse in that aspect, so given that btrfs saved my bacon with a dying ssd, I can only imagine what zfs does.
You can certainly add verification above and below your filesystem, but the filesystem seems like a good layer to have verification. Capturing a checksum while writing and verifying it while reading seems appropriate; zfs scrub is a convenient way to check everything on a regular basis. Personally, my data feels important enough to make that level of effort, but not important enough to do anything else.
*Coming from the extremely well thought out and documented zfs utilities to btrfs will have you wondering wtf fairly frequently while you learn your way around.
The dm stuff is one key for the entire partition and you can't check it for bitrot or repair it without the key.
> Personally, my data feels important enough to make that level of effort, but not important enough to do anything else.
OMG. Backups! You need backups! Worry about polishing your geek cred once your data is on physically separate storage. Seriously, this is not a technology choice problem. Go to Amazon and buy an exfat stick, whatever. By far the most important thing you're ever going to do for your data is Back. It. Up.
Filesystem choice is, and I repeat, very much a yell-on-the-internet kind of thing. It makes you feel smart on HN. Backups to junky Chinese flash sticks are what are going to save you from losing data.
I wouldn't comment but I feel like I'm naturally on your side of the argument and want to see it articulated well.
A block level cache like bcache (not fs) and dm-cache handles it less ideally, and doesn't leave the SSD space as usable space. As a home user, 2TB of SSDs is 2TB of space I'd rather have. ZFS's ZIL is similar, not leaving it as usable space. Btrfs has some recent work in differentiating drives to store metadata on the faster drives (allocator hints), but that only does metadata as there is no handling of moving data to HDDs over time. Even Microsoft's ReFS does tiered storage I believe.
I just want to have 1 or 2 SSDs, with 1 or 2 HDDs in a single filesystem that gets the advantages of SSDs with recently used files and new writes, and moves all the LRU files to the HDDs. And probably keep all the metadata on the SSDs too.
My goal was actually the same though: to try to short-circuit the inevitable platform flame by calling it out explicitly and pointing out that the technical details are sort of a solved problem.
ZFS argumentation gets exhausting, and has ever since it was released. It ends up as a proxy for Sun vs. Linux, GNU vs. BSD, Apple vs. Google, hippy free software vs. corporate open source, pick your side. Everyone has an opinion, everyone thinks it's crucially important, and as a result of that hyperbole everyone ends up thinking that ZFS (dtrace gets a lot of the same treatment) is some kind of magically irreplaceable technology.
And... it's really not. Like I said above if it disappeared from the universe and everyone had to use dm/lvm for the actual problems they need to solve with storage management[1], no one would really care.
[1] Itself an increasingly vanishing problem area! I mean, at scale and at the performance limit, virtually everything lives behind a cloud-adjacent API barrier these days, and the backends there worry much more about driver and hardware complexity than they do about mere "filesystems". Dithering about individual files on individual systems in the professional world is mostly limited to optimizing boot and update time on client OSes. And outside the professional world it's a bunch of us nerds trying to optimize our movie collections on local networks; realistically we could be doing that on something as awful NTFS if we had to.
I spent some time researching this topic, and in all benchmarks I've seen and my personal tests btrfs is faster or much faster: https://www.reddit.com/r/zfs/comments/1i3yjpt/very_poor_perf...
https://wiki.archlinux.org/title/Dm-integrity
> It uses journaling for guaranteeing write atomicity by default, which effectively halves the write speed
I'd really rather not do that, thanks.
IIRC my laptop's zpool has a 1.2x compression ratio; it's worth doing. At a previous job, we had over a petabyte of postgres on ZFS and saved real money with compression. Hilariously, on some servers we also improved performance because ZFS could decompress reads faster than the disk could read.
1. This is misunderstanding how device corruption works. It's not and can't ever be limited to "files". (Among other things: you can lose whole trees if a directory gets clobbered, you'd never even be able to enumerate the "corrupted files" at all!). All you know (all you can know) is that you got a success and that means the relevant data and metadata matched the checksums computed at write time. And that property is no different with dm. But if you want to know a subset of the damage just read the stderr from tar, or your kernel logs, etc...
2. Metadata robustness in the face of inconsistent updates (e.g. power loss!) is a feature provided by all modern filesystems, and ZFS is no more or less robust than ext4 et. al. But all such filesystems (ZFS included) will "lose data" that hadn't been fully flushed. Applications that are sensitive to that sort of thing must (!) handle this by having some level of "transaction" checkpointing (i.e. a fsync call). ZFS does absolutely nothing to fix this for you. What is true is that an unsynchronized snapshot looks like "power loss" at the dm level where it doesn't in ZFS. But... that's not useful for anyone that actually cares about data integrity, because you still have to solve the power loss problem. And solving the power loss problem obviates the need for ZFS.
But I don't usually verify the backups, so there's that. And everything is in the same zip code for the most part, so one big disaster and I'll lose everything. C'est la vie.
In the future bcachefs will be rolling out auxiliary dirent indices for a variety of purposes, and one of those will be to give you a list of files that have had errors detected by e.g. scrub (we already generally tell you the affected filename in error messages)
2 - No, metadata robustness absolutely varies across filesystems.
From what I've seen, ext4 and bcachefs are the gold standard here; both can recover from basically arbitrary corruption and have no single points of failure.
Other filesystems do have single points of failure (notably btree roots), and btrfs and I believe ZFS are painfully vulnerable to devices with broken flush handling. You can blame (and should) blame the device and the shitty manufacturers, but from the perspective of a filesystem developer, we should be able to cope with that without losing the entire filesystem.
XFS is quite a bit better than btrfs, and I believe ZFS, because they have a ton of ways to reconstruct from redundant metadata if they lose a btree root, but it's still possible to lose the entire filesystem if you're very, very unlucky.
On a modern filesystem that uses b-trees, you really need a way of repairing from lost b-tree roots if you want your filesystem to be bulletproof. btrfs has 'dup' mode, but that doesn't mean much on SSDs given that you have no control over whether your replicas get written to the same erase unit.
Reiserfs actually had the right idea - btree node scan, and reconstruct your interior nodes if necessary. But they gave that approach a bad name; for a long time it was a crutch for a buggy b-tree implementation, and they didn't seed a filesystem specific UUID into the btree node magic number like bcachefs does, so it could famously merge a filesystem from a disk image with the host filesystem.
bcachefs got that part right, and also has per-device bitmaps in the superblock for 'this range of the device has btree nodes' so it's actually practical even if you've got a massive filesystem on spinning rust - and it was introduced long after the b-tree implementation was widely deployed and bulletproof.
based on my own testing, dm has a lot of footguns and, with some kernels, as little as 100 bytes of corruption to the underlying disk could render a dm-integrity volume completely unusable (requiring a full rebuild) https://github.com/khimaros/raid-explorations
As I understand it ZFS also has a lot of redundant metatdata (copies=3 on anything important), and also previous uberblocks[1].
In what way is XFS better? Genuine question, not really familiar with XFS.
[1]: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSMetadata...
I do a ton of reading through forums gathering user input, and lots of people chime in with stories of lost filesystems. I've seen reports of lost filesystems with ZFS and I want to say I've seen them at around the same frequency of XFS; both are very rare.
My concern with ZFS is that they seem to have taken the same "no traditional fsck" approach as btrfs, favoring entirely online repair. That's obviously where we all want to be, but that's very hard to get right, and it's been my experience that if you prioritize that too much you miss the "disaster recovery" scenarios, and that seems to be what's happened with ZFS; I've read that if your ZFS filesystem is toast you need to send it to a data recovery service.
That's not something I would consider acceptable, fsck ought to be able to do anything a data recovery service would do, and for bcachefs it does.
I know the XFS folks have put a ton of outright paranoia into repair, including full on disaster recovery scenarios. It can't repair in scenarios where bcachefs can - but on the other hand, XFS has tricks that bcachefs doesn't, so I can't call bcachefs unequivocally better; we'd need to wait for more widespread usage and a lot more data.
But there's a ton of room for improvement beyond what ZFS did. ZFS was a very conservative design in a lot of ways (rightly so! so many ambitious projects die because of second system syndrome); notably, it's block based and doesn't do extents - extents and snapshots are a painfully difficult combination.
Took me years to figure that one out.
My hope for bcachefs has always been to be a real successor to ZFS, with better and more flexible management, better performance, and even better robustness and reliability.
Long road, but the work continues.
Say more? I can't say I've really thought that much about filesystems and I'm curious in what direction you think they could be taken if time and budget weren't an issue.
This is my favorite side effect of compression in the right scenarios. I remember getting a huge speed up in a proprietary in-memory data structure by using LZO (or one of those fast algorithms) which outperformed memcpy, and this was already in memory so no disk io involved! And used less than a third of the memory.
Maybe, if you never create anything. I make a lot of game art source and much of that is in uncompressed formats. Like blend files, obj files, even DDS can compress, depending on the format and data, due to the mip maps inside them. Without FS compression it would be using GBs more space.
I'm not going to individually go through and micromanage file compression even with a tool. What a waste of time, let the FS do it.
It's an entirely clean slate design, and I spent years taking my time on the core planning out the design; it's as close to perfect as I can make it.
The only things I can think of that I would change or add given unlimited time and budget: - It should be written in Rust, and even better a Rust + dependent types (which I suspect could be done with proc macros) for formal verification. And cap'n proto for on disk data structures (which still needs Rust improvements to be as ergonomic as it should be) would also be a really nice improvement.
- More hardening; the only other thing we're lacking is comprehensive fault injection testing of on disk errors. It's sufficiently battle hardened that it's not a major gap, but it really should happen at some point.
- There's more work to be done in bitrot prevention: data checksums really need to be plumbed all the way into the pagecache
I'm sure we'll keep discovering new small ways to harden, but nothing huge at this point.
Some highlights: - It has more defense in depth than any filesystem I know of. It's as close to impossible to have unrecoverable data loss as I think can really be done in a practical production filesystem - short of going full immutable/append only.
- Closest realization of "filesystem as a database" that I know of
- IO path options (replication level, compression, etc.) can be set on a per file or directory basis: I'm midway through a project extending this to do some really cool stuff, basically data management is purely declarative.
- Erasure coding is much more performant than ZFS's
- Data layout is fully dynamic, meaning you can add/remove devices at will, it just does the right thing - meaning smoother device management than ZFS
- The way the repair code works, and tracking of errors we've seen - fantastic for debugability
- Debugability and introspection are second to none: long bug hunts really aren't a thing in bcachefs development because you can just see anything the system is doing
There's still lots of work to do before we're fully at parity with ZFS. Over the next year or two I should be finishing erasure coding, online fsck, failure domains, lots more management stuff... there will always be more cool projects just over the horizon
I will not use or recommend ZFS on _any_ OS until they solve the double page cache problem. A filesystem has no business running its own damned page cache that duplicates the OS one. I don't give a damn if ZFS has a fancy eviction algorithm. ARC's patent is expired. Go port it to mainline Linux if it's not that good. Just don't make inner platform.
You were not alone. However, things changed, namely SSD continued to become cheaper and grew in capacity. I'd think most active data is these days on SSDs (certainly in most desktops, most servers which aren't explicit file or DB servers and all mobile and embedded devices), the role of spinning rust being more and more archiving (if found in a system at all).
I obviously have nothing like inside knowledge, but I assume the reason there have not been lawsuits over this, is that whoever could bring one (would it be only Oracle?) expects there are even-odds that they would lose? Thus the risk of setting an adverse precedent isn't worth the damages they might be awarded from suing Canonical?
But Sun ensured that they can only gnash their teeth.
The source of "license incompatibility" btw is the same as from using GPLv3 code in kernel - CDDL adds an extra restriction in form of patent protections (just like Apache 2)
For low-level meddling and recovery, there's a filesystem debugger that understands all parts of ZFS and can help for example identifying previous uberblock that is uncorrupted, or recovering specific data, etc.
It's extremely well thought out, it's minimalist in all the right ways; I've found the features and optimizations it has to be things that are borne out of real experience that you would want end up building yourself in any real world system.
E.g. it gives you the ability to add new fields without breaking compatibility. That's the right way to approach forwards/backwards compatibility, and it's what I do in bcachefs and if we'd been able to just use cap'n proto it would've taken out a lot of manual fiddly work.
The only blocker to using it more widely in my own code is that it's not sufficiently ergonomic in Rust - Rust needs lenses, from Swift.
any plans for much lower rates than typical raid?
Increasingly modern high density devices are having block level failures at non-trivial rates instead of or in addition to whole device failures. A file might be 100,000 blocks long, adding 1000 blocks of FEC would expand it 1% but add tremendous protection against block errors. And can do so even if you have a single piece of media. Doesn't protect against device failures, sure, though without good block level protection device level protection is dicey since hitting some block level error when down to minimal devices seems inevitable and having to add more and more redundant devices is quite costly.
I am not a lawyer.
My understanding is that Canonical is shipping ZFS with Ubuntu. Or do I misunderstand? Has Canonical not actually done the big, bad thing of distributing the Linux kernel with ZFS? Did they find some clever just-so workaround so as to technically not be violation of the Linux kernel's license terms?
Otherwise, if Canonical has actually done the big, bad thing, who has standing to bring suit? Would the Linux Foundation sue Canonical, or would Oracle?
I ask this in all humility, and I suspect there is a chance that my questions are nonsense and I don't know enough to know why.
If there's an optimized implementation we can use in the kernel, I'd love to add it. Even on modern hardware, we do see bit corruption in the wild, it would add real value.
I was more thinking along the lines of adding dozens or hundreds of correction blocks to a whole file, along the lines of par (though there are much faster techniques now).
What happens on ZFS if you lose all your alloc info? Or are there other single points of failure besides the ublock in the on disk format?
I think SSDs are generally worse than spinning rust (especially enterprise grade SCSI kit), the hard drive vendors have been at this a lot longer and SSDs are massively more complicated. From the conversations I've had with SSD vendors, I don't think they've put the some level of effort into making things as bulletproof as possible yet.
More so than BFS?
https://en.m.wikipedia.org/wiki/Be_File_System
"Like its predecessor, OFS (Old Be File System, written by Benoit Schillings - formerly BFS), it includes support for extended file attributes (metadata), with indexing and querying characteristics to provide functionality similar to that of a relational database."
Generally a code that can always detect N errors can only always correct N/2 errors. So you detect an errored block, you correct up to N/2 errors. The block now passes but if the block actually had N errors, your correction will be incorrect and you now have silent corruption.
The solution to this is just to have an excess of error correction power and then don't use all of it. But that can be hard to do if you're trying to shoehorn it into an existing 32-bit crc.
How big are the blocks that the CRC units cover in bcachefs?
But I'm talking more about the internals than external database functionality; the inner workings are much more fundamental.
bcachefs internally is structured more like a relational database than a traditional Unix filesystem, where everything hangs off the inode. In bcachefs, there's an extents btree (read: table), an inodes btree, a dirents btree, and a whole bunch of others - we're up to 20 (!).
There's transactions, where you can do arbitrary lookups, updates, and then commit, with all the database locking hidden from you; lookups within a transaction see uncommitted updates from that transaction. There's triggers, which are used heavily.
We don't have the full relational model - no SELECT or JOIN, no indices on arbitrary fields like with SQL (but you can do effectively the same thing with triggers, I do it all the time).
All the database/transactional primitives make the rest of the codebase much smaller and cleaner, and make feature development a lot easier than what you'd expect in other filesystems.
"Note that increasing iodepth beyond 1 will not affect synchronous ioengines"[1]
Is there a reason you used that ioengine as opposed to, for example, "libaio" with a "--direct=1" flag?
[1] https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-...
On a different note, have you heard about prolly trees and structural sharing? It’s a newer data structure that allows for very cheap structural sharing and I was wondering if it would be possible to build an FS on top of it to have a truly distributed fs that can sync across machines.
This is a really good tradeoff in practice; the vast majority of applications are doing buffered IO, not small block O_DIRECT reads - that really only comes up in benchmarks :)
And it gets us better compression ratios and better metadata overhead.
We also have quite a bit of flexibility to add something bigger to the extent for FEC, if we need to - we're not limited to a 32/64 bit checksum.
Hopefull we can get to a point where Linux has a native, and first-class modern alternative to ZFS with BcacheFS.
A ZFS pool will remain available even in degraded mode, and correct me if I'm wrong but with BTRFS you mount the array through one of the volume that is part of the array and not the array itself.. so if that specific mounted volume happens to go down, the array becomes unavailable unmounted until you remount another available volume that is part of the array which isn't great for availability.
I thought about mitigating that by making an mdadm RAID1 formatted with BTRFS and mount the virtual volume instwad, but then you lose the ability to prevent bit rot, since BTRFS lose that visibility if it doesn't manage the array natively.
According to this[1] old issue, it hasn't happened frequently enough to prioritize implementing a rebuild option, however one should be able to import the pool read-only and zfs send it to a different pool.
As far as I can tell that's status quo. I agree it is something that should be implemented at some point.
That said, certain other spacemap errors might be recoverable[2].
[1]: https://github.com/openzfs/zfs/issues/3210
[2]: https://github.com/openzfs/zfs/issues/13483#issuecomment-120...
Yes. Oracle have that copyright.
That's the whole fucking point.
Anything from before the fork is still licensed (and pretty much everything after) is still under the CDDL which is possibly in conflict with the GPL.
It would be an interesting lawsuit as the judge might well ask why as copyright holder of ZFS they can't solve the problem they are suing over. But I think you underestimate the deviousness of oracle's legal dept.
If I see an issue that causes a filesystem to become unavailable _once_, I'll write the repair code.
Experience has taught me that there's a good chance I'll be glad I did, and I like the peace of mind that I get from that.
And it hasn't been that bad to keep up on, thanks to lucky design decisions. Since bcachefs started out as bcache, with no persistent alloc info, we've always had the ability to fully rebuild alloc info, and that's probably the biggest and hardest one to get right.
You can metaphorically light your filesystem on fire with bcachefs, and it'll repair. It'll work with whatever is still there and get you a working filesystem again with the minimum possible data loss.
The CDDL being unacceptable is the same issue that GPL3 or Apache is unacceptable - unlike GPLv2, CDDL mandates patent licensing as far as the code is considered.
Additionally, GPLv2 does not prevent shipping ZFS combined with GPL code, because CDDL code is not derivative work of GPLv2 code. So it's legal to ship.
It could be problematic to upstream, because kernel development would demand streamlining to the point that the code would be derivative.
Additionally, two or three kernel contributors decided that the long standing consensus on derivative work is not correct and sued Canonical. So far nothing happened out of that, Los Alamos National Laboratory also laughed it off.
Venue shopping being what it is, though...
As I mentioned in the other subthread, I do think your commitment to help your users is very commendable.
* read-only and minimal
* fully aware of different Linux boot environments
* GPLv3 license compatible, clean-room implementation by the OpenSolaris/Illumos team. The implementation predates Ubuntu’s interest.The CDDL code is not a derivative work of GPLv2 code, but the combined work as a whole is a derivative work of GPLv2 code (assuming by "combined" we are talking about shipping an executable made by compiling and linking GPLv2 and CDDL code together). Shipping that work does require permission from both the GPLv2 code copyright owners and the CDDL code copyright owners unless the code from on or the other can be justified under fair use or if it was a part of the GPLv2 or CDDL code that is not subject to copyright.
What Canonical does is ship ZFS as a kernel module. That contains minimal GPLv2 code from the kernel that should be justifiable as fair use (which seems like a decent bet after the Oracle vs Google case).
Just trying to give honest assessments and comparisons.
The CDDL-only parts of the driver are portable between OSes, removing the "derivative code" argument (similar argumentation goes back to introduction of AFS driver for Linux, IIRC).
Remember, GPLv2 does not talk about linking. Derivativeness is decided by source code, among other things whether or not the non-GPL code can't exist/operate without GPL code.
A lot of people focus on the fact that the CDDL allows binaries to be arbitrarily licensed so long as you provide the sources, but the issue is that the GPL requires that the source code of both combined and derived works be under the GPL and the CDDL requires that the source code be under the CDDL (i.e., the source code cannot be sublicensed). This means that (if you concluded that OpenZFS is a derived work of Linux or that it is a combined work when shipped as a kernel module) a combination may be a violation of both licenses.
However, the real question is whether a judge would look at two open source licenses that are incompatible due to a technicality and would conclude that Oracle is suffering actual harm (even though OpenZFS has decades of modifications from the Oracle version). They might also consider that Oracle themselves released DTrace (also under the CDDL) for their Linux distribution in 2012 as proof that Oracle doesn't consider it to be license violation either. If we did see Canonical get sued, maybe we'd finally be able to find out through discovery if the CDDL was intentionally designed to be GPL incompatible or not (a very contentious topic).
[1]: https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/
Unlike the GPL, the CDDL (and MPL) has an opt-out upgrade clause and all of OpenSolaris (or more accurately, almosf all software under the CDDL) can be upgraded to "CDDL-1.1 OR CDDL-2.0" unilaterally by Oracle even if they do not own the copyrights. See section 4 of the CDDL.
I don't think btrfs has a concept of having only some subvolumes usable. Either you can mount the filesystem or you can't. What may have confused you is that you can mount a btrfs filesystem by referring to any individual block device that it uses, and the kernel will track down the others. But if the one device you have listed in /etc/fstab goes missing, you won't be able to mount the filesystem without fixing that issue. You can prevent the issue in the first place by identifying the filesystem by UUID instead of by an individual block device.
AFAIU, btrfs effectively absolves itself of responsibility in these cases, claiming the issue is buggy drive firmware.
1) Making CDDL compatible with GPLv2 puts everyone using CDDL code at mercy of Oracle patents
2) OpenZFS is actually not required to upgrade, and the team has indicated they won't. So you end up with a fork you need to carry yourself. Might even force OpenZFS to ensure that it's specifically 1.0.
Ultimately it means Oracle can't do much with this.
Also for video games, where performance matters, game sizes are huge, and it's nice to have a bunch of games installed.
1) They could just adapt MPL-2.0, which provides GPLv2+ compatibility while still providing the same patent grants.
2) The upgrade is chosen by downstream users. The OpenZFS project could ask individual contributiors to choose to license their future contributions differently but that will only affect future versions and isn't a single decision made by the project leads. I don't know in what context that discussion was in but given that the have not already opted-out of future CDDL versions kind of indicates that they can imagine future CDDL versions they would choose to upgrade to.
Also, OpenZFS is under CDDL-1.1 not 1.0.