←back to thread

197 points slaily | 1 comments | | HN request time: 0.203s | source
Show context
slaily ◴[] No.44530519[source]
If you’re building Python async apps (FastAPI, background jobs, etc.) with SQLite, you’ll eventually hit two issues

- Opening/closing connections is fast, but not free—overhead adds up under load

- SQLite writes are globally locked

aiosqlitepool is a tiny library that adds connection pooling for any asyncio SQLite driver (like aiosqlite):

- It avoids repeated database connection setup (syscalls, memory allocation) and teardown (syscalls, deallocation) by reusing long-lived connections

- Long-lived connections keep SQLite's in-memory page cache "hot." This serves frequently requested data directly from memory, speeding up repetitive queries and reducing I/O operations

- Allows your application to process significantly more database queries per second under heavy load

Enjoy!

replies(5): >>44565059 #>>44565071 #>>44566400 #>>44570162 #>>44570945 #
slashdev ◴[] No.44565059[source]
How does this help with the second issue, the write locks?
replies(1): >>44565309 #
ncruces ◴[] No.44565309[source]
No idea if it applies, but one way would be to direct all writes (including any transaction that may eventually write) to a single connection.

Then writers queue up, while readers are unimpeded.

replies(1): >>44567014 #
dathinab ◴[] No.44567014[source]
if you enable WAL mode with sqlite then readers are not blocked by writer so only writers queue up without needing any special case handling to archive it

(in general you _really_ should use WAL mode if using sqlite concurrently, you also should read the documentation about WAL mode tho)

replies(3): >>44568037 #>>44568286 #>>44570395 #
1. le-mark ◴[] No.44570395[source]
WAL doesn’t cure concurrency issues for SQLite. WAL plus single writer, multiple reader threaded is required. It’s blazing fast though.