As a professional rust dev, I will say this, I don't notice. Generally because I am doing partial builds mostly. And when I am not the m1 max I have is fast enough to compile the project.
Really there are much larger issues that I have to deal with that build time does not affect.