←back to thread

Futurelock: A subtle risk in async Rust

(rfd.shared.oxide.computer)
427 points bcantrill | 4 comments | | HN request time: 0.653s | source

This RFD describes our distillation of a really gnarly issue that we hit in the Oxide control plane.[0] Not unlike our discovery of the async cancellation issue[1][2][3], this is larger than the issue itself -- and worse, the program that hits futurelock is correct from the programmer's point of view. Fortunately, the surface area here is smaller than that of async cancellation and the conditions required to hit it can be relatively easily mitigated. Still, this is a pretty deep issue -- and something that took some very seasoned Rust hands quite a while to find.

[0] https://github.com/oxidecomputer/omicron/issues/9259

[1] https://rfd.shared.oxide.computer/rfd/397

[2] https://rfd.shared.oxide.computer/rfd/400

[3] https://www.youtube.com/watch?v=zrv5Cy1R7r4

Show context
jacquesm ◴[] No.45776483[source]
If any rust designers are lurking about here: what made you decide to go for the async design pattern instead of the actor pattern, which - to me at least - seems so much cleaner and so much harder to get wrong?

Ever since I started using Erlang it felt like I finally found 'the right way' when before then I did a lot of work with sockets and asynchronous worker threads. But even though it usually worked as advertised it had a large number of really nasty pitfalls which the actor model seemed to - effortlessy - step aside.

So I'm seriously wondering what the motivation was. I get why JS uses async, there isn't any other way there, by the time they added async it was too late to change the fundamentals of the language to such a degree. But rust was a clean slate.

replies(5): >>45776498 #>>45776569 #>>45776637 #>>45776798 #>>45777596 #
sunshowers ◴[] No.45776498[source]
Not a Rust designer, but a big motivation for Rust's async design was wanting it to work on embedded, meaning no malloc and no threads. This unfortunately precludes the vast majority of the design space here, from active futures as seen in JS/C#/Go to the actor model.

You can write code using the actor model with Tokio. But it's not natural to do so.

replies(5): >>45776630 #>>45776683 #>>45777051 #>>45777075 #>>45778490 #
oconnor663 ◴[] No.45777051[source]
Kind of a tangent, but I think "systems programming" tends to bounce back and forth between three(?) different concerns that turn out to be closely related:

1. embedded hardware, like you mentioned

2. high-performance stuff

3. "embedding" in the cross-language sense, with foreign function calls

Of course the "don't use a lot of resources" thing that makes Rust/C/C++ good for tiny hardware also tends to be helpful for performance on bigger iron. Similarly, the "don't assume much about your runtime" thing that's necessary for bare metal programming also helps a lot with interfacing with other languages. And "run on a GPU" is kind of all three of those things at once.

So yeah, which of those concerns was async Rust really designed around? All of them I guess? It's kind of like, once you put on the systems programming goggles for long enough, all of those things kind of blend together?

replies(1): >>45778676 #
kibwen ◴[] No.45778676[source]
> So yeah, which of those concerns was async Rust really designed around? All of them I guess?

Yes, all of them. Futures needed to work on embedded platforms (so no allocation), needed to be highly optimizable (so no virtual dispatch), and need to act reasonably in the presence of code that crosses FFI boundaries (so no stack shenanigans). Once you come to terms with these constraints--and then add on Rust's other principles regarding guaranteed memory safety, references, and ownership--there's very little wiggle room for any alternative designs other than what Rust came up with. True linear types could still improve the situation, though.

replies(2): >>45778767 #>>45778958 #
mjevans ◴[] No.45778767[source]
In my view, the major design sin was not _forcing_ failure into the outcome list.

.await(DEADLINE) (where deadline is any non 0 unit, and 0 is 'reference defined' but a real number) should have been the easy interface. Either it yields a value or it doesn't, then the programmer has to expressly handle failure.

Deadline would only be the minimum duration after which the language, when evaluating the future / task, would return the empty set/result.

replies(2): >>45779231 #>>45781407 #
1. danielheath ◴[] No.45779231[source]
Does that imply a lot of syscalls to get the monotonic clock value? Or is there another way to do that?
replies(2): >>45779274 #>>45780705 #
2. mjevans ◴[] No.45779274[source]
If the scheduler is doing _any_ sort of accounting at all to figure out any remote sort of fairness balancing at all, then whatever resolution that is probably works.

At least for Linux, offhand, popular task scheduler frequencies used to be 100 and 1000hz.

Looks like the Kernel's tracking that for tasks:

https://www.kernel.org/doc/html/latest/scheduler/sched-desig...

"In CFS the virtual runtime is expressed and tracked via the per-task p->se.vruntime (nanosec-unit) value."

I imagine the .vruntime struct field is still maintained with the newer "EEVDF Scheduler".

...

A Userspace task scheduler could similarly compare the DEADLINE against that runtime value. It would still reach that deadline after the minimum wait has passed, and thus be 'background GCed' at a time of the language's choice.

replies(1): >>45779597 #
3. jitl ◴[] No.45779597[source]
The issue is that no scheduler manages futures. The scheduler sees tasks, futures are just a struct. See discussion of embedded above: there is no “kernel esque” parallel thread
4. muvlon ◴[] No.45780705[source]
On Linux there is the VDSO, which on all mainstream architectures allows you to do `clock_gettime` without going through a syscall. It should take on the order of (double digit) nanoseconds.