←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 4 comments | 19 Nov 24 00:15 UTC | HN request time: 0.634s | source
Show context
WiSaGaN ◴[19 Nov 24 02:12 UTC] No.42179482[source]▶
I am wondering how much cost is needed for serving at such a latency. Of course for customers, static cost depends on the pricing strategy. But still, the cost really determines how widely this can be adopted. Is it only for those business that really need the latency, or this can be generally deployed.
replies(2): >>42179566 #>>42179803 #
1. ilaksh ◴[19 Nov 24 03:18 UTC] No.42179803[source]▶
Maybe it could become standard for everyone to make giant chips and use SRAM?

How many SRAM manufacturers are there? Or does it somehow need to be fully integrated into the chip?

replies(2): >>42179856 #>>42180718 #
2. AlotOfReading ◴[19 Nov 24 03:28 UTC] No.42179856[source]▶
SRAM is usually produced on the same wafer as the rest of the logic. SRAM on an external chip would lose many of the advantages without being significantly cheaper.
replies(1): >>42180689 #
3. YetAnotherNick ◴[19 Nov 24 06:57 UTC] No.42180689[source]▶
Yes, the limiting factor for bandwidth is generally the number of pins which are not cheap and you can only have few 1000s in a chip. The absolute state of the art is 36 Gb/s/pin[1], and your $30 RAM could have 6 Gb/s/pin[2].

[1]: https://en.wikipedia.org/wiki/GDDR7_SDRAM

[2]: https://en.wikipedia.org/wiki/DDR5_SDRAM

4. why_only_15 ◴[19 Nov 24 07:02 UTC] No.42180718[source]▶
the cost is not the memory technology per se but primarily the wires. SRAM is fast because it's directly inside the chip and so the connections with the logic that does the work is cheap because it's close.