←back to thread

Building a Durable Execution Engine with SQLite

(www.morling.dev)
110 points ingve | 1 comments | 20 Nov 25 13:26 UTC | HN request time: 0.001s | source
Show context
websiteapi ◴[21 Nov 25 19:09 UTC] No.46007761[source]▶
there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here?

you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.

replies(5): >>46007901 #>>46008038 #>>46008154 #>>46008279 #>>46009559 #
1. ryeats ◴[21 Nov 25 19:32 UTC] No.46008038[source]▶
As you say it can be done but it's an anti-pattern to use a message queue as a database which is essentially what you are doing for these kinds of long running tasks. The reason is that their are a lot of state your likely going to want to status as a task runs and persist and checkpoint yes you can carefully string together a series of database calls chained with message transactions so you don't lose something when an issue happens but then you also need bespoke logic to restart or retry each step and it can turn into a bit of a mess.