Sqlite3 WebAssembly

Author here. If you're interested, do ask questions.

Very cool project! Do you know if this would be possible for duckdb? Is there something about sqlites APIs and wasm build that made it feasible?

Context: Currently using go-duckdb and while it's working for us, getting rid of cgo would be a huge help. Would be quite interested myself to attempt this.

I don't know much about DuckDB's architecture.

Wasm is fine for compute (though concurrency is still a somewhat open question).

To have Wasm talk to the outside world, you need “host calls” where the guest calls the host.

On a browser that's Wasm calling JavaScript. On my Go driver, it's Wasm calling Go.

For server side, there's also a standard set of “host calls” modeled around POSIX/Linux syscalls called WASI.

I could've build my project around WASI, but WASI is rather limited (and SQLite support for WASI was more limited even, it's improved a bit since). DuckDB might work out-of-the-box this way.

I, instead, took advantage of SQLite's architecture and replaced its VFS layer with one in Go: https://sqlite.org/vfs.html

So SQLite in Wasm is just doing compute, and I do all the OS level stuff in Go. No need for Wasm concurrency, cause I can load multiple instances of my Wasm which act like independent OS processes that communicate through the filesystem (SQLite excels at this).

As I said, I dunno how well all those decisions would map to DuckDB.

> So SQLite in Wasm is just doing compute, and I do all the OS level stuff in Go. No need for Wasm concurrency, cause I can load multiple instances of my Wasm which act like independent OS processes that communicate through the filesystem (SQLite excels at this).

Interesting. So when I am running concurrent readers using your package, it is just loading multiple instances of the wasm code? (I bottleneck to a single writer in the application)

Hey how's the mmap-based shared memory WAL approach working out? It's been about half a year since you finished the implementation and I only see one issue about it on the tracker, a good sign?

It's turning out great, I guess.

The current approach is not portable to Windows, but it works fine on Linux, macOS, BSD and illumos. In general, portability is hindered more by file locking (I hate POSIX locks) than mmap.

The currently open GitHub issue is more bad default configuration than anything else. Configuring connections to use less memory by default should fix it.

I already have a PR ready for the next release that also opens this up for 32-bit platforms.

Also, GoToSocial (a self-hostable Mastodon alternative) moved to it (from modernc) for its first beta release.

https://github.com/superseriousbusiness/gotosocial

Yes.

Each connection lives in its own isolated sandbox, and only communicates with other connections through the “file system” (which is a virtual abstraction, actually).

WAL mode is the “exception”: a few pages of the sandbox's memory are mapped to a file, and shared by all connections to the same database.

Each sandbox is single threaded, and mostly lock free, does all its business in the calling goroutine, and regularly checks back with the Go runtime to play nice with the Go scheduler.

It's a bit like an OS running multiple processes, with the VFS layer handling all syscalls.

This is why I've switched to go and am never looking back.