Sqlite3 WebAssembly

So after downloading from the official downloads page and stripping away all the mjs files and "bundler-friendly" files, a minimal sqlite wasm dependency will be about 1.3MB.

For an in-browser app, that seems a bit much but of course wasm runs in other places these days where it might make more sense.

The thing to keep in mind is that the WebAssembly sandbox model means that in theory the program (SqlLite in this case) can run wherever it makes sense to run it. That might mean running it locally or it might mean running on a central server or it might mean running nearby on the “edge”.

It's a good consideration, together with the fact browsers already have IndexedDB embedded. One use case still for in-browser apps like Figma / Photoshop-like / ML apps, where the application code and data is very big anyway, 1.3Mb may not add that much

Also worth considering parsing of wasm is significantly faster than JS (unfortunately couldn't find the source for this claim, there is at lease one great article on the topic)

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_A...

It's pretty compressible at least, sqlite3.js+wasm are 1.3MB raw but minifying the JS and then compressing both files with Brotli gets them down to 410KB.

1.3MB seems perfectly reasonable in a modern web app, especially since it will be cached after the first visit to the site.

If you’re just storing user preferences, obviously don’t download SQLite for your web app just to do that… but if you’re doing something that benefits from a full database, don’t fret so much about 1MB that you go try to reinvent the wheel for no reason.

If the other comment is correct, then it won’t even be 1.3MB on the network anyways.

When we built our frontend sync system we tried a few different options. We had a fairly simple case of just trying to store entities so we could pull incremental updates since you were last online. The one we ran in production for a while was IndexedDB but found the overhead wasn’t worth it.

I played around with warm sqlite too. That was really nice but I decided against it due to the fact that it was totally unsupported.

For server side, you'll likely need a different build of Wasm SQLite, that handles concurrency (and file locking) differently.

Also, WASI is very far from answer (so far). The SQLite amalgamation builds fine for WASI but concurrency is an unsolved issue.

I had to build a VFS from scratch to get my Wasm based SQLite driver into a usable shape.

https://github.com/ncruces/go-sqlite3/blob/main/vfs/README.m...

A megabyte here, a megabyte there, pretty soon you’re talking about a really heavyweight app.

Is there a way to statically compile an application with SQLite and the result WASM was smaller. So for example I have an app that would use only a specific subset of SQLite. Could the SQLite's WASM be built with this in mind cutting down on code that is not used? Or is there a way to prune it having the used API surface?

In a regular compiler/linker scenario it would just be a static compilation. Here we have a JS app and WASM library.

Since SQL takes arbitrary strings as input, this would require explicit compiler flags to disable the knobs you don't want. Can't rely on excluding unused symbols really.

Given how hefty images are, a full database doesn't seem too bad for the purpose of an "app" that would benefit from it, especially when compression can being the size down even lower.

We are past the stage where every piece of JS has to be loaded upfront and delay the first meaningful paint. Modern JS frameworks and module are chunked and can be eager/lazy loaded. Unless you make the sqlite DB integral part for your first meaningful page load, preloading those 1.3MB in the background/upon user request is easy.

A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!).

Adding 400 for such a high quality piece of DB actually borders reasonability.

And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.

> what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable

They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.

Sadly, 1.3 MB is nothing on the modern web, especially for a static file. BBC's frontpage loads 3.78 MB.

https://www.bbc.co.uk/

> A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!).

I think you can probably blame Tailwind for that.

> BBC's frontpage loads 3.78 MB.

FWIW: Google Drive just downloaded 15.4mb to boot up for me and imdb dot com hit some 7+mb before it started auto-loading videos on top of that.

> Could the SQLite's WASM be built with this in mind cutting down on code that is not used?

The pending 3.47 release has some build-side tweaks which enable a user to strip it down to "just the basics," but we've not yet been able to get it smaller than about 25-30% less than it otherwise is:

    cd ext/wasm
    make barebones=1 ; # requires GNU Make and the Emscripten SDK

Doing that requires building it yourself - there are no plans to publish deliverables built that way.

The build process also supports including one's own C code, which could hypothetically be used to embed an application and the wasm part of the library (as distinct from the JS part) into a single wasm file. Its primary intended usage is to add SQLite extensions which are not part of the standard amalgamation build.

> Or is there a way to prune it having the used API surface?

Not with the provided JS pieces. Those have to expose essentially the whole C library, so they will not be pruned from the wasm file.

However, you could provide your own JS bindings which only use a small subset of the API, and Emscripten is supposedly pretty good about stripping out C-side code which neither explicitly exported nor referenced anywhere. You'd be on your own - that's not something we'll integrate into the canonical build process - but we could provide high-level support, via the project's forum, for folks taking that route.

That's correct, people in this thread are comparing single compressed dependency of sqlite+wasm of 400KB to the total size of web pages which run in MB. I did some actual tests while trying to use sqlite and it does adds noticeable delay on first page load on mobile due to big size+decompression+ additional scaffolding of wasm. Pages that run into MB have small files that are downloaded concurrently so the delay is not noticeable. I wrote about this and my other expriments with in browser db in my last article but it did not get any traction here.

> They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.

Context makes all the difference here. If you're considering a big chunk of size for a relational database engine, you need to ask: are you making a complex application, or a normal web page? If it's the latter, then it's not reasonable at all.

And anything that makes the HTML itself that big is almost certainly bloat, not "high quality", and shouldn't be used in any context.

By the time you have a good reason to add this library, I think you're already in heavyweight app territory.

Why? More often than not the classes are combined during post-processing to the most reusable unified classes, with very short classes names.

You're comparing 2 Mb of useless and broken animated scrolling with 400 Kb of SQLite, which tells me you have no idea what's behind SQLite's 400 kb.

High quality software is something I rarely see nowadays when browsing "modern" websites.

Correction:

    make barebones=1 ; # requires GNU Make and the Emscripten SDK

should be:

    make oz barebones=1 ; # requires GNU Make and the Emscripten SDK

otherwise it will build with -O0, resulting in huge wasm files.