My take on Rust build times is that the larger issue isn't the speed of compilation itself, but that it's actually pretty hard to avoid compiling a lot of unneeded code from dependencies. To be clear, this is a distinct concern from having too many dependencies (which is it's own issue, but I'd argue that it's a social one that doesn't need to be solved in order to significantly reduce build times, which can be done with a technical solution).
I wrote up a lot of my ideas in a blog post mostly to avoid having to repeat my thoughts on this every time it comes up: https://saghm.com/cargo-features-rust-compile-times/