←back to thread

Sqlite3 WebAssembly

(sqlite.org)
647 points whatever3 | 2 comments | | HN request time: 0.41s | source
Show context
simonw ◴[] No.41851934[source]
Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects (https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st...) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too).

Then on subsequent visits use that server-side data to rehydrate the client-side database.

From https://sqlite.org/forum/info/50a4bfdb294333eec1ba4749661934... is looks like WAL mode is excluded from the default SQLite WASM build so you would have to go custom with that.

replies(10): >>41852040 #>>41852194 #>>41853089 #>>41854540 #>>41854586 #>>41854654 #>>41855596 #>>41856415 #>>41857000 #>>41858635 #
ncruces ◴[] No.41853089[source]
There are many layers of that's not how it works at play here.

In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode.

Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm).

Cloudflare Durable Objects may have been inspired by Litestream but works very differently (as do LiteFS, Turso, etc…)

The general idea of streaming changes from SQLite would work, but it's a lot of work, and the concurrency model of in-browser Wasm will make it challenging to implement.

(I wrote that forum post some time ago, and have WAL working in a server side Wasm build of SQLite, but none of the options to make it work would make much sense, or be possible, in browser)

replies(5): >>41853607 #>>41853645 #>>41853823 #>>41857460 #>>41860639 #
1. rockwotj ◴[] No.41857460[source]
I believe that streaming changes from SQLite is what https://sqlsync.dev/ is
replies(1): >>41857829 #
2. ncruces ◴[] No.41857829[source]
Yep, that doesn't use WAL mode at all, and it's a decent amount of work.

It uses a custom VFS, memory journal mode, and bypasses SQLite for optimistic concurrency handling. I dunno how it handles crash safety, or if it even uses the same on disk file format.

My point here is not that this is impossible, it's that SQlite's WAL implementation isn't meant to be pluggable, and Litestream is a very clever hack, that requires a very specific setup to work well.

It takes a fair amount of work to replicate that in other environments.