←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 1 comments | 19 Nov 24 00:15 UTC | HN request time: 0.218s | source
Show context
LASR ◴[19 Nov 24 02:25 UTC] No.42179539[source]▶
What you can do with current-gen models, along with RAG, multi-agent & code interpreters, the wall is very much model latency, and not accuracy any more.

There are so many interactive experiences that could be made possible at this level of token throughput from 405B class models.

replies(2): >>42179814 #>>42191188 #
TechDebtDevin ◴[19 Nov 24 03:20 UTC] No.42179814[source]▶
Like what..
replies(2): >>42180079 #>>42180145 #
1. davidfiala ◴[19 Nov 24 04:21 UTC] No.42180079[source]▶
Imagine increasing the quality and FPS of those AI-generated minecraft clones and experiencing even more high-quality, realtime AI-generated gameplay

(yeah, I know they are doing textual tokens. but just sayin..)

edit: context is https://oasisaiminecraft.com/