←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 1 comments | 19 Nov 24 00:15 UTC | HN request time: 0.211s | source
Show context
WiSaGaN ◴[19 Nov 24 02:12 UTC] No.42179482[source]▶
I am wondering how much cost is needed for serving at such a latency. Of course for customers, static cost depends on the pricing strategy. But still, the cost really determines how widely this can be adopted. Is it only for those business that really need the latency, or this can be generally deployed.
replies(2): >>42179566 #>>42179803 #
ilaksh ◴[19 Nov 24 03:18 UTC] No.42179803[source]▶
Maybe it could become standard for everyone to make giant chips and use SRAM?

How many SRAM manufacturers are there? Or does it somehow need to be fully integrated into the chip?

replies(2): >>42179856 #>>42180718 #
1. why_only_15 ◴[19 Nov 24 07:02 UTC] No.42180718[source]▶
the cost is not the memory technology per se but primarily the wires. SRAM is fast because it's directly inside the chip and so the connections with the logic that does the work is cheap because it's close.