←back to thread

Litestream VFS

(fly.io)
306 points emschwartz | 5 comments | 11 Dec 25 17:59 UTC | HN request time: 0.194s | source
Show context
orliesaurus ◴[11 Dec 25 19:06 UTC] No.46235678[source]▶
Been tinkering with litestream... the read-only VFS is neat but I'm curious about eventual write capabilities... using VFS for distributed DBs could unlock some interesting patterns.

ALSO I'm thinking about mixing this with object store caching... maybe combining memfs with remote metadata; would love to see more details on performance.

BUT I might be overthinking it... just excited to see SQLite exploring beyond local files...

replies(1): >>46235849 #
1. reactordev ◴[11 Dec 25 19:17 UTC] No.46235849[source]▶
Opens up a whole new can of worms. Transactions come to mind. Who would be responsible for coordinating? If two nodes wrote to the table conflicting information at the same time, who wins?
replies(1): >>46236349 #
2. benbjohnson ◴[11 Dec 25 19:59 UTC] No.46236349[source]▶
Author here. We've done some proof-of-concept work on creating distributed leases using S3. We have some use cases internally where we've considered adding write capabilities to the VFS but we haven't started any work on it yet.
replies(2): >>46236976 #>>46237404 #
3. hobo_mark ◴[11 Dec 25 20:52 UTC] No.46236976[source]▶
Is this VFS for read-only databases? Or can I query a database that has a single litestream writer somewhere continously making updates and backing them up to S3?
replies(1): >>46237275 #
4. benbjohnson ◴[11 Dec 25 21:15 UTC] No.46237275{3}[source]▶
The VFS is read only but it will continuously poll for new updates so if you have a writer somewhere else using regular Litestream then it will pick up those updates automatically.
5. reactordev ◴[11 Dec 25 21:28 UTC] No.46237404[source]▶
It’s a tricky problem that goes beyond the fs as you know. Since it’s cloud and since it’s distributed, a manager wouldn’t be that far fetched that could issue CRDT like messages across the cluster of nodes to issue a “write” or utilize the sync mechanism you have to propagate a “master” db that you write to (aggregate or designate, either way). I did some work on this on a go based database graph and ended up doing a gossip sync crdt message bus.