I don't really consider it to be slow at all. It seems about as performant as any other language this complexity, and it's far faster than the 15 minute C++ and Scala build times I'd place in the same category.
I also don’t understand this, the rust compiler hardly bothers me at all when I’m working. I feel like this is due to how bad it was early on and people just sticking to that narrative