←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 3 comments | 07 Sep 23 14:36 UTC | HN request time: 0.819s | source
Show context
superkuh[dead post] ◴[07 Sep 23 15:33 UTC] No.37420475[source]▶
[flagged]
sbierwagen ◴[07 Sep 23 15:34 UTC] No.37420490[source]▶
M2 Mac Studio with 192gb of ram is US$5,599 right now.
replies(3): >>37420616 #>>37420693 #>>37427799 #
superkuh[dead post] ◴[07 Sep 23 15:45 UTC] No.37420693[source]▶
[flagged]
1. piskov ◴[07 Sep 23 15:47 UTC] No.37420725[source]▶
You do understand that you can connect thunderbolt external storage (not just usb3 one)?
replies(1): >>37420784 #
2. diffeomorphism ◴[07 Sep 23 15:50 UTC] No.37420784[source]▶
That does not really make 1tb of non-upgradable storage in a $5k+ device any less ridiculous though.
replies(1): >>37420864 #
3. yumraj ◴[07 Sep 23 15:55 UTC] No.37420864[source]▶
That is true, but a whole separate discussion.

It applies to RAM too. My 32GB Mac Studio seemed pretty good before the LLMs.