←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0s | source
Show context
superkuh[dead post] ◴[07 Sep 23 15:33 UTC] No.37420475[source]▶
[flagged]
YetAnotherNick ◴[07 Sep 23 16:00 UTC] No.37420966[source]▶
You just need 4 3090($4000) to run it. And 4 3090 are definitely lot more powerful and versatile than an M2 mac.
replies(3): >>37421025 #>>37421444 #>>37424271 #
yumraj ◴[07 Sep 23 16:04 UTC] No.37421025[source]▶
How much would that system cost, if you could easily buy those GPUs
replies(2): >>37421217 #>>37421964 #
1. iaw ◴[07 Sep 23 16:59 UTC] No.37421964[source]▶
I build one on 4 used $800 3090s. All told $6-6.5K in used components to build an open air rig with consumer hardware.

I went with the cheapest threadripper I could find but after testing the performance hit for going from 16-8 PCIe lanes was actually not that large and I would be okay going with 8 lanes for each card.