They could have used SDRAM and it would perform so much better, and I believe the cost is around the same.
If you wanted to cut something, cut the antialiasing. While very cool, it is a bit wasted on CRTs. Worst of all, for some reason they have this blur filter which smears the picture horizontally. Luckily it can be deblured by appliying the inverse operation.
By the time the N64 launched, SDRAM was better and cheaper, and they considered it was too late to make the switch. Allegedly SGI wanted to make changes but Nintendo refused.
Basically they made the wrong bet and didn't want to change it closer to release.
OK, I also just read that basically Nintendo bet on ram bandwidth, but ignored latency.
A more general lesson: Nintendo bet on cutting edge, speculative technology with RDRAM, instead of concentrating on 'Lateral Thinking with Withered Technology'.