←back to thread

Building a Durable Execution Engine with SQLite

(www.morling.dev)
110 points ingve | 1 comments | 20 Nov 25 13:26 UTC | HN request time: 0s | source
Show context
websiteapi ◴[21 Nov 25 19:09 UTC] No.46007761[source]▶
there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here?

you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.

replies(5): >>46007901 #>>46008038 #>>46008154 #>>46008279 #>>46009559 #
1. tptacek ◴[21 Nov 25 19:22 UTC] No.46007901[source]▶
We build what is effectively a durable execution "engine" for our orchestrator (ours is backed by boltdb and not SQLite, which I objected to, correctly). The steps in our workflows build running virtual machines and include things like allocating addresses, loading BPF programs, preparing root filesystems, and registering services.

Short answer: we need to be able to redeploy and bounce the orchestrator without worrying about what stage each running VM on our platform is in.

JP, the dev that built this out for us, talks a bit about the design rationale (search for "Cadence") here:

https://fly.io/blog/the-exit-interview-jp/

The library itself is open:

https://github.com/superfly/fsm