←back to thread

197 points slaily | 2 comments | | HN request time: 0s | source
Show context
slaily ◴[] No.44530519[source]
If you’re building Python async apps (FastAPI, background jobs, etc.) with SQLite, you’ll eventually hit two issues

- Opening/closing connections is fast, but not free—overhead adds up under load

- SQLite writes are globally locked

aiosqlitepool is a tiny library that adds connection pooling for any asyncio SQLite driver (like aiosqlite):

- It avoids repeated database connection setup (syscalls, memory allocation) and teardown (syscalls, deallocation) by reusing long-lived connections

- Long-lived connections keep SQLite's in-memory page cache "hot." This serves frequently requested data directly from memory, speeding up repetitive queries and reducing I/O operations

- Allows your application to process significantly more database queries per second under heavy load

Enjoy!

replies(5): >>44565059 #>>44565071 #>>44566400 #>>44570162 #>>44570945 #
slashdev ◴[] No.44565059[source]
How does this help with the second issue, the write locks?
replies(1): >>44565309 #
ncruces ◴[] No.44565309[source]
No idea if it applies, but one way would be to direct all writes (including any transaction that may eventually write) to a single connection.

Then writers queue up, while readers are unimpeded.

replies(1): >>44567014 #
dathinab ◴[] No.44567014{3}[source]
if you enable WAL mode with sqlite then readers are not blocked by writer so only writers queue up without needing any special case handling to archive it

(in general you _really_ should use WAL mode if using sqlite concurrently, you also should read the documentation about WAL mode tho)

replies(3): >>44568037 #>>44568286 #>>44570395 #
rich_sasha ◴[] No.44568286{4}[source]
I was running into some horrendous issues with WAL, where the WAL file would grow boundlessly, eventually leading to veery slow reads and writes.

It's fixable by periodically forcing the WAL to be truncated, but it took me a lot of time and pain to figure it out.

replies(2): >>44568711 #>>44572720 #
1. dathinab ◴[] No.44572720{5}[source]
The is why I said read the WAL doc page in a different answer ;)

They do point out the risks here: https://sqlite.org/wal.html#avoiding_excessively_large_wal_f...

sqlites design makes a lot of SQL concurrency synchronization edge cases much simpler as you can rely on the single writer at a time limitation. And it has some grate hidden features for using it as client application state storage. But there are use-cases it's just not very good at and moving from sqlite to other DBs can be tricky (if you ever relied on the exclusive write transaction or the way cells are blobs which can mix data types, even it it was by accident)

replies(1): >>44573110 #
2. rich_sasha ◴[] No.44573110[source]
I did read it. For whatever reason, automatic checkpoints basically would stop from time to time, and the WAL file would start growing like crazy.

In the end I wrote an external process that forced a checkpoint a few times a day, which worked. I came across other exasperated people in various dark corners of the Internet with the same symptoms.