Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

To be clear a cerebras chip is consuming a whole wafer and has only 44 GB of SRAM on it. To fit a 405B model in bf16 precision (excluding kv cache and activation memory usage) you need 19 of these “chips” (and the requirement will grow as the sequence length increases for the kvcache). Looking online it seems on one wafer one can fit between 60 to 80 H100 chips, so it’s equivalent to using >1500 H100 using wafer manufacturing cost as a metric

The budget these companies spend on this tech is seriously mind boggling to me.

Is wafer cost a major factor in the actual chip price?