←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.212s | source
Show context
tiffanyh ◴[07 Sep 23 16:11 UTC] No.37421142[source]▶

  system_info: n_threads = 4 / 24
Am I seeing correctly in the video that this ran on only 4 threads?
replies(1): >>37422416 #
1. wmf ◴[07 Sep 23 17:25 UTC] No.37422416[source]▶
It's using the GPU so I guess not that many CPU threads are needed to feed the GPU.