←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 3 comments | 19 Nov 24 00:15 UTC | HN request time: 0.542s | source
Show context
zackangelo ◴[19 Nov 24 02:11 UTC] No.42179476[source]▶
This is astonishingly fast. I’m struggling to get over 100 tok/s on my own Llama 3.1 70b implementation on an 8x H100 cluster.

I’m curious how they’re doing it. Obviously the standard bag of tricks (eg, speculative decoding, flash attention) won’t get you close. It seems like at a minimum you’d have to do multi-node inference and maybe some kind of sparse attention mechanism?

replies(9): >>42179489 #>>42179493 #>>42179501 #>>42179503 #>>42179754 #>>42179794 #>>42180035 #>>42180144 #>>42180569 #
simonw ◴[19 Nov 24 04:08 UTC] No.42180035[source]▶
They have a chip the size of a dinner plate. Take a look at the pictures: https://cerebras.ai/product-chip/
replies(3): >>42180116 #>>42180150 #>>42180490 #
1. Aeolun ◴[19 Nov 24 04:44 UTC] No.42180150[source]▶
21 petabytes per second. Can push the whole internet over that chip xD
replies(2): >>42180630 #>>42180744 #
2. why_only_15 ◴[19 Nov 24 06:44 UTC] No.42180630[source]▶
The number for that is I believe 1 terabit or 125GB/s -- 21 petabytes is the speed from the SRAM (~registers) to the cores (~ALU) for the whole chip. It's not especially impressive for SRAM speeds. The impressive thing is that they have an enormous amount of SRAM
3. KeplerBoy ◴[19 Nov 24 07:08 UTC] No.42180744[source]▶
That's their on chip cache bandwidth. Usually that stuff isn't even measured in bandwidth but latency.