←back to thread

What If OpenDocument Used SQLite?

(www.sqlite.org)
275 points whatisabcdefgh | 5 comments | 04 Sep 25 21:36 UTC | HN request time: 0s | source
Show context
liuliu ◴[05 Sep 25 00:24 UTC] No.45133756[source]▶
One thing I would call out, if you use SQLite as an application format:

BLOB type is limited to 2GiB in size (int32). Depending on your use cases, that might seem high, or not.

People would argue that if you store that much of binary data in a SQLite database, it is not really appropriate. But, application format usually has this requirement to bundle large binary data in one nice file, rather than many files that you need to copy together to make it work.

replies(1): >>45134101 #
1. Retr0id ◴[05 Sep 25 01:18 UTC] No.45134101[source]▶
You can split your data up across multiple blobs
replies(3): >>45135219 #>>45136032 #>>45140501 #
2. johncolanduoni ◴[05 Sep 25 05:24 UTC] No.45135219[source]▶
Also you almost certainly want to do this anyway so you can stream the blobs into/out of the network/filesystem, well before you have GBs in a single blob.
replies(1): >>45136933 #
3. bob1029 ◴[05 Sep 25 07:48 UTC] No.45136032[source]▶
This is essential if you want to have encryption/compression + range access at the same time.

I've been using chunk sizes of 128 megabytes for my media archive. This seems to be a reasonable tradeoff between range retrieval delay and per object overhead (e.g. s3 put/get cost).

4. Retr0id ◴[05 Sep 25 10:16 UTC] No.45136933[source]▶
Singular sqlite blobs are streamable too! But for streaming in you need to know the size in advance.
5. liuliu ◴[05 Sep 25 16:34 UTC] No.45140501[source]▶
That's right, but it is much easier to just use blob without application logic to worry about chunking. It is the same reason why we use SQLite in the first place, a lot of transaction / rollback logic now is on SQLite layer, not the application layer.

Also, SQLite did provide good support for read / write the blob in streamable fashion, see: https://www.sqlite.org/c3ref/blob_read.html

So the limitation is really a structural issue that Dr. Hipp at some point might resolve (or not), but pretty much has to be resolved by SQLite core team, not outside contributors (of course you can resolve it by forking, but...).