Of course, Nintendo clearly cared about the CPU a lot for marketing purposes (it's in the console's name), but from a purely technological perspective, it is wasteful. Most of the actual compute is done on the RSP anyway. So, getting a much smaller CPU would have been a big corner to cut, that could have saved enough resources to increase the texture cache to a useful resolution like 128x128 or so.
It should be noted, though, that the N64 was designed with multitexturing capabilities, which would have helped with the mushy colors had games actually taken advantage of it (but they didn't, which here again, the Nintendo SDK is to blame for).
Only really in the marketing material. It's a bit like calling a 386 with an arithmetic co-processor an 80 bit machine, when it was still clearly a 32 bit machine by all metrics that matter.
However, I agree in general that the N64 CPU sits idle a lot of the time. It's overspecced compared to the rest of the system.