←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.203s | source
1. homarp ◴[07 Sep 23 15:16 UTC] No.37420186[source]▶
https://www.reddit.com/r/LocalLLaMA/comments/16bynin/falcon_... has some more data like sample answers with various level of quantizations

and https://huggingface.co/TheBloke/Falcon-180B-Chat-GGUF if you want to try