←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.201s | source
Show context
superkuh[dead post] ◴[07 Sep 23 15:33 UTC] No.37420475[source]▶
[flagged]
sbierwagen ◴[07 Sep 23 15:34 UTC] No.37420490[source]▶
M2 Mac Studio with 192gb of ram is US$5,599 right now.
replies(3): >>37420616 #>>37420693 #>>37427799 #
superkuh[dead post] ◴[07 Sep 23 15:45 UTC] No.37420693[source]▶
[flagged]
1. ErneX ◴[07 Sep 23 16:51 UTC] No.37421852[source]▶
You can plug an NVME thunderbolt caddy instead, it won’t reach a good NVME SSD top speeds but it will hover around 2800MB/s r+w.

Its internal SSD at 1TB or greater capacity is at least twice as fast.