It was clear they were trying to do more consumer grade things, just look at the N64. Couldn't get more mainstream than that. Seeing how the graphics market ended up, it looks obvious from here but in the mid 90's it was still the wild west and everybody was throwing mud at the wall seeing what would stick.
I have never really said that they where "taken by surprise", but a part of it felt like (from the outside) that management had been a little blinded by their pass success and the profit margins from their workstations combined with no clear path forwards for the whole industry. Nvidia could have very easily been just a curiosity of the past but they managed to strike it lucky standing on the shoulders of others.
If SGI had always been a company that could provide graphics workstations the worked with x86/Windows PC's early for example - maybe they would have fared better. Would have gone with the flow of technology at the time rather than fighting uphill no matter the potential technical brilliance. But being saddled to their MIPS processors and custom OS meant that once people left, they almost never came back. One can have the best tech and still fail.