←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 8 comments | 19 Nov 24 00:15 UTC | HN request time: 0.425s | source | bottom
1. brcmthrowaway ◴[19 Nov 24 03:03 UTC] No.42179727[source]▶
So out of all AI chip startups, Cerebras is probably the real deal
replies(2): >>42179835 #>>42179935 #
2. gdiamos ◴[19 Nov 24 03:23 UTC] No.42179835[source]▶
just in time for their ipo
replies(1): >>42179878 #
3. ipsum2 ◴[19 Nov 24 03:31 UTC] No.42179878[source]▶
It got cancelled/postponed.
4. icelancer ◴[19 Nov 24 03:46 UTC] No.42179935[source]▶
Groq is legitimate. Cerebras so far doesn't scale (wide) nearly as good as Groq. We'll see how it goes.
replies(2): >>42180141 #>>42180942 #
5. hendler ◴[19 Nov 24 04:41 UTC] No.42180141[source]▶
Google TPUs, Amazon, a YC funded ASIC/FPGA company, a Chinese Co. all have custom hardware too that might scale well.
6. throwawaymaths ◴[19 Nov 24 07:55 UTC] No.42180942[source]▶
How exactly does groq scale wide well? Last I heard it was 9 racks!! to run llama-2 70b

Which is why they throttle your requests

replies(1): >>42187154 #
7. pama ◴[19 Nov 24 19:28 UTC] No.42187154{3}[source]▶
Well, Cerebras pretty much needs a data center to simply fit the 405B model for inference.
replies(1): >>42187359 #
8. throwawaymaths ◴[19 Nov 24 19:47 UTC] No.42187359{4}[source]▶
I guess this just shows the insanity of venture led AI hardware hype and shady startup messaging practices