←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.349s | source
Show context
superkuh[dead post] ◴[07 Sep 23 15:33 UTC] No.37420475[source]▶
[flagged]
YetAnotherNick ◴[07 Sep 23 16:00 UTC] No.37420966[source]▶
You just need 4 3090($4000) to run it. And 4 3090 are definitely lot more powerful and versatile than an M2 mac.
replies(3): >>37421025 #>>37421444 #>>37424271 #
yumraj ◴[07 Sep 23 16:04 UTC] No.37421025[source]▶
How much would that system cost, if you could easily buy those GPUs
replies(2): >>37421217 #>>37421964 #
PartiallyTyped ◴[07 Sep 23 16:14 UTC] No.37421217[source]▶
Lanes will probably be an issue, so a threadripper pro or an epyc cpu, add half a grand at least for the motherboard and it’s starting to look grim.
replies(1): >>37421346 #
thfuran ◴[07 Sep 23 16:22 UTC] No.37421346[source]▶
And that's before you even get your first power bill.
replies(2): >>37421385 #>>37422182 #
1. PartiallyTyped ◴[07 Sep 23 16:24 UTC] No.37421385[source]▶
hey, at least you will cut down on the heating costs!