←back to thread

Building a Durable Execution Engine with SQLite

(www.morling.dev)
110 points ingve | 3 comments | 20 Nov 25 13:26 UTC | HN request time: 0.408s | source
Show context
websiteapi ◴[21 Nov 25 19:09 UTC] No.46007761[source]▶
there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here?

you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.

replies(5): >>46007901 #>>46008038 #>>46008154 #>>46008279 #>>46009559 #
1. snicker7 ◴[21 Nov 25 19:46 UTC] No.46008154[source]▶
Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as:

- Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag)

You really do need a database for this.

replies(2): >>46009178 #>>46009553 #
2. yyx ◴[21 Nov 25 21:27 UTC] No.46009178[source]▶
Sounds like a Celery with SQLAlchemy backend.
3. DenisM ◴[21 Nov 25 22:07 UTC] No.46009553[source]▶
I’m reminded of classical LRU cache implementation - double linked list and a hash map that points to the list elements.

It is a queue if we squint really hard, but it allows random access and reordering. Do we have durable structures of this kind?

I can’t imagine how to shoehorn this into Kafka or SQS.