←back to thread

Futurelock: A subtle risk in async Rust

(rfd.shared.oxide.computer)
421 points bcantrill | 4 comments | | HN request time: 0.659s | source

This RFD describes our distillation of a really gnarly issue that we hit in the Oxide control plane.[0] Not unlike our discovery of the async cancellation issue[1][2][3], this is larger than the issue itself -- and worse, the program that hits futurelock is correct from the programmer's point of view. Fortunately, the surface area here is smaller than that of async cancellation and the conditions required to hit it can be relatively easily mitigated. Still, this is a pretty deep issue -- and something that took some very seasoned Rust hands quite a while to find.

[0] https://github.com/oxidecomputer/omicron/issues/9259

[1] https://rfd.shared.oxide.computer/rfd/397

[2] https://rfd.shared.oxide.computer/rfd/400

[3] https://www.youtube.com/watch?v=zrv5Cy1R7r4

Show context
hitekker ◴[] No.45777349[source]
Skimming through, this document feels thorough and transparent. Clearly, a hard lesson learned. The footnotes, in particular, caught my eye https://rfd.shared.oxide.computer/rfd/397#_external_referenc...

> Why does this situation suck? It’s clear that many of us haven’t been aware of cancellation safety and it seems likely there are many cancellation issues all over Omicron. It’s awfully stressful to find out while we’re working so hard to ship a product ASAP that we have some unknown number of arbitrarily bad bugs that we cannot easily even find. It’s also frustrating that this feels just like the memory safety issues in C that we adopted Rust to get away from: there’s some dynamic property that the programmer is responsible for guaranteeing, the compiler is unable to provide any help with it, the failure mode for getting it wrong is often undebuggable (by construction, the program has not done something it should have, so it’s not like there’s a log message or residual state you could see in a debugger or console), and the failure mode for getting it wrong can be arbitrarily damaging (crashes, hangs, data corruption, you name it). Add on that this behavior is apparently mostly undocumented outside of one macro in one (popular) crate in the async/await ecosystem and yeah, this is frustrating. This feels antithetical to what many of us understood to be a core principle of Rust, that we avoid such insidious runtime behavior by forcing the programmer to demonstrate at compile-time that the code is well-formed

replies(2): >>45778263 #>>45779161 #
csande17 ◴[] No.45779161[source]
In case anyone else was confused: the link/quote in this comment are from the previous "async cancellation issue" write-up, which describes a situation where you "drop" a future: the code in the async function stops running, and all the destructors on its local variables are executed.

The new write-up from OP is that you can "forget" a future (or just hold onto it longer than you meant to), in which case the code in the async function stops running but the destructors are NOT executed.

Both of these behaviors are allowed by Rust's fairly narrow definition of "safety" (which allows memory leaks, deadlocks, infinite loops, and, obviously, logic bugs), but I can see why you'd be disappointed if you bought into the broader philosophy of Rust making it easier to write correct software. Even the Rust team themselves aren't immune -- see the "leakpocalypse" from before 1.0.

replies(3): >>45780439 #>>45780713 #>>45780756 #
formerly_proven ◴[] No.45780439[source]
async rust continues to strike me as half-baked and too complex, if you’re developing an application (as opposed to some high performance utility like e.g. a data plane component) just use threads, they’re plenty cheap and not even half as messy.
replies(3): >>45781030 #>>45783781 #>>45784006 #
1. kibwen ◴[] No.45781030[source]
Async Rust is as complex as it needs to be given its constraints. But I wholeheartedly agree with you that people need to treat threads (especially scoped ones) as the default concurrency primitive. My intuition is that experience with other languages has led people astray; in most languages threads are a nightmare and/or async is the default or only way to achieve concurrency, but threads in Rust are absolutely divine by comparison. Async should only be used when you have a good reason that threads don't suffice.
replies(2): >>45781321 #>>45782204 #
2. redman25 ◴[] No.45781321[source]
It's a good idea in concept but tons of popular libraries use async which makes it difficult to avoid. Want to do anything with a web server or sending requests, most likely async for popular libraries.
replies(1): >>45781538 #
3. galangalalgol ◴[] No.45781538[source]
Yeah, the nom asynch nats client got deprecated for instance. It really is a shame, because very few projects will ever scale large enough to need asynch, and apart from things like this, there are costs in portability and supply chain attack surface area when you bring in tokio.
4. mwcampbell ◴[] No.45782204[source]
In the spirit of "every non-trivial program will expand until ...", I think preemptively choosing async for anything much more complex than a throwaway script might be justified. In this case, the relevant thing isn't performance or expected number of concurrent users/connections, but whether the program is likely to become or include a non-trivial state machine. My primary influence on this topic is this post from @sunshowers: https://sunshowers.io/posts/nextest-and-tokio/