←back to thread

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference

(cerebras.ai)
426 points benchmarkist | 1 comments | 19 Nov 24 00:15 UTC | HN request time: 0.21s | source
Show context
owenpalmer ◴[19 Nov 24 06:33 UTC] No.42180575[source]▶
The fact that such a boost is possible with new hardware, I wonder what the ceiling is for improving performance for training via hardware as well.
replies(2): >>42180618 #>>42180710 #
bufferoverflow ◴[19 Nov 24 06:42 UTC] No.42180618[source]▶
The ultimate solution would be to convert an LLM to a pure ASIC.

My guess is that would 10X the performance. But then it's a very very expensive solution.

replies(2): >>42180716 #>>42187711 #
1. tiagod ◴[19 Nov 24 20:25 UTC] No.42187711[source]▶
There's some interesting research on using stacked flat lenses to build analog, physical neural network inference that operate directly on light (each lens is a hidden layer). If we managed to make this work for non-trivial cases, it could be absurdly fast.