Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

nvidia hates this one little trick

I laughed and upvoted, but if anything I bet they put their best people on it to replicate this offering.

What I take away from this is: we are just getting started. I remember in 2023 begging OpenAI to give us more than 7 tokens/second on GPT-4.

Nvidia’s target is performance across concurrent users and they are likely already outperforming Cerebras there as far as costs are concerned. They have no reason to try to beat the single user performance of this.