←back to thread

Litestream VFS

(fly.io)
306 points emschwartz | 2 comments | 11 Dec 25 17:59 UTC | HN request time: 0.001s | source
Show context
jauntywundrkind ◴[11 Dec 25 18:42 UTC] No.46235250[source]▶
So much fun streaming/sync/cdc stuff happening, all so cool. Having an underlying FUSE driver doing the Change Data Capture is really neat. This looks like such an incredibly lightweight way to remote-connect to sqlite. And to add a sort of exterior transaction management.

Different use case, but makes me think of sqlite Rewrite-it-it-Rust Turso announcing AgentFS. Here the roles are flipped, sqlite is acting as a file store to back FUSE, to allow watching/transaction-managing the filesystem/what agents are doing. Turso also has a sick CDC system built in, that just writes all changes to a cdc table. Which is related to this whole meta question, of what is happening to my sqlite DB. https://turso.tech/blog/agentfs

replies(1): >>46235261 #
tptacek ◴[11 Dec 25 18:43 UTC] No.46235261[source]▶
Just to be clear, the underlying FUSE thing is LiteFS, not Litestream; nothing described in this post needs a FUSE filesystem, just a small SQLite VFS plugin library.
replies(1): >>46235371 #
1. jauntywundrkind ◴[11 Dec 25 18:49 UTC] No.46235371[source]▶
Thanks Thomas.

To just drop the relevant paragraph that addresses my un-clarity/in-correctness (and which is super fun to read):

> Litestream v0.5 integrates LTX, our SQLite data-shipping file format. Where earlier Litestream blindly shipped whole raw SQLite pages to and from object storage, LTX ships ordered sets of pages. We built LTX for LiteFS, which uses a FUSE filesystem to do transaction-aware replication for unmodified applications, but we’ve spent this year figuring out ways to use LTX in Litestream, without all that FUSE drama.

replies(1): >>46235418 #
2. tptacek ◴[11 Dec 25 18:52 UTC] No.46235418[source]▶
You got it. Yeah, Ben built LiteFS a year or two ago, which is why he designed LTX. But using LiteFS required people to set up FUSE filesystems, which was too complicated for a lot of people, so Litestream got way more uptake. This past year he's been harvesting all the good stuff from LiteFS that doesn't require FUSE and building it into Litestream.

The easiest way so far to understand the split between Litestream and LiteFS: Litestream is an operational tool, for backup and restore. LiteFS is a method for doing online leader/follower replica clusters.