←back to thread

65 points qvr | 1 comments | | HN request time: 0.211s | source
Show context
nullc ◴[] No.44653420[source]
Are any filesystems offering file level FEC yet?

If a file has a hundred thousand blocks you could tack on a thousands blocks of error correction for the cost of making it just 1% larger. If the file is a seldom/never written archive it's essentially free beyond the space it takes up.

The kind of massive data archives that you want to minimize storage costs of tend to be read-mostly affairs.

It won't save you from a disk failure but I see bad blocks much more often than whole disk failures these days... and raid/5/6 have rather high costs while being still quite vulnerable to the possibility of an aligned fault on multiple disks.

Of course you could use par or similar tools, but that lacks nice FS transparent integration and particularly doesn't benefit from checksums already implemented in (some) FS (as you need half the error correction data to recover from known-position errors, and-or can use erasure only codes).

replies(5): >>44653469 #>>44653485 #>>44654527 #>>44655069 #>>44655131 #
IgnaciusMonk ◴[] No.44654527[source]
newer, smaller (physically,epitaxy, not capacity) ssd "cell" = more times per year you have to rewrite(refresh) that cell / whole disk so you do not lose data, anyway.

any sane person uses FS / system with dedup in it, so you can have 7+4+12 snapshots for 5TB of data taking only 7TB of space. etc

you want snapshots, for example Manjarolinux (arch based) does use BTRFS capable of snapshots. so before every update it will make snapshot so if update fails, you can just select to go back into working state in grub...

Alma linux uses BTRFS too but im not sure if they have this functionality too.

ZFS, bcache, BTRFS, checksums

MD-INTERGITY inside of linux kernel can provide checksums for any fs essentially. just " lvcreate --type raidN --raidintegrity y " and you have checksumms + raid in linux

replies(1): >>44654612 #
1. IgnaciusMonk ◴[] No.44654612[source]
without unraid / nonraid nonsense. Just " pure Linux™ " Trademark owned by Linus Torvalds

filesystem scrubbing.