Unexpected productivity boost of Rust

Imagine if every thread tries to acquire this mutex on every HTTP request. And then you also lock it in a spawn_blocking. On a heavily contended lock, all the reactor threads will end up stuck waiting for the spawn_blocking to finish and the service will appear hung since even requests that don’t even need that lock can’t get processed. And if spawn_blocking is doing synchronous network I/O or reading a file from a hung NFS mount, you’re completely hosed.

You would be in a similarly bad situation if such a lock was async as well. You want to keep locks as short as possible, regardless, and a hung task with a lock is generally going to propagate issues throughout the system until it's dealt with.

If you have an async lock that can be held across await points, your motivation to spawn_block decreases. Additionally, even though the long running lock would be a problem, APIs that aren't taking that lock continue to work vs the entire HTTP service being hung which is a huge difference (e.g. health monitoring would time out making the service seem unresponsive vs 1 highly contended task just waiting on a lock).

>e.g. health monitoring would time out making the service seem unresponsive vs 1 highly contended task just waiting on a lock

If anything that's a disadvantage. You want your health monitoring to be the canary, not something that keeps on trucking even if the system is no longer doing useful work. (See the classic safety critical software fail of 'I need a watchdog... I'll just feed it regularly in an isolated task')

I disagree and hope I can convince you. You're assuming that any request has an equal probability of causing this problem so it's kind of OK if the server deadlocks under load so that the failing health check triggers a restart by whatever is watching it. However, let's say you have three API endpoints:

    /healthz
    /very_common_operation
    /may_deadlock_server

Normally, /may_deadlock_server doesn't get enough traffic to cause problems (let's say it's 10 RPS and 1000 RPS is /very_common_operation and the server operates fine). However, a sudden influx of requests to /may_deadlock_server may cause your service to deadlock (and not a lot, let's say on the order of a few hundred requests). Do you still want the server to lock up completely and forever and wait for a healthz timeout to reboot the service? What if healthz still remains fine but the entire service goes from 10ms response times for requests to 200ms, just enough to cause problems but not enough to make healthz actually unavailable? And all this just because /may_deadlock saw a spike in traffic. And also, the failing healthz check just restarts your service but it won't mitigate the traffic spike if it's sustained. Now consider also that /may_deadlock_server is a trivial gadget for an attacker to DOS your site.

Or do you want the web server responding healthily & rely on metrics and alerts to let you know that /may_deadlock_server is taking a long time to handle requests / impacting performance? Your health monitoring is an absolute last step for automatically mitigating an issue but it'll only help if the bug is some state stuck in a transient state - if it'll restart into the same conditions leading to the starvation then you're just going to be in an infinite reboot loop which is worse.

Healthz is not an alternative to metrics and alerting - it's a last stopgap measure to try to automatically get out of a bad situation. But it can also cause a worse problem if the situation is outside of the state of the service - so generally you want the service to remain available if a reboot wouldn't fix the problem.