←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.239s | source
Show context
pella ◴[07 Sep 23 15:20 UTC] No.37420249[source]▶
Is this an M2 Ultra with 192 GB of unified memory, or the standard version with 64 GB of unified memory?
replies(1): >>37420741 #
1. zargon ◴[07 Sep 23 15:48 UTC] No.37420741[source]▶
4-bit quantized 180B will not fit in 64GB. You'll need need over 100 GB for that.