←back to thread

Running a 180B parameter LLM on a single Apple M2 Ultra

(twitter.com)
255 points tbruckner | 2 comments | 07 Sep 23 14:36 UTC | HN request time: 0.443s | source
Show context
randomopining ◴[07 Sep 23 15:40 UTC] No.37420598[source]▶
Is there any actual usecases to run this stuff on a local computer? Or are most of these models actually suited to run on remote clusters?
replies(3): >>37421417 #>>37421858 #>>37423070 #
acdha ◴[07 Sep 23 16:51 UTC] No.37421858[source]▶
Here’s a simple one: corporate policy doesn’t allow you to send company data to a cloud service. There are a ton of people with significant budgets in that situation.
replies(1): >>37424659 #
1. zamadatix ◴[07 Sep 23 19:51 UTC] No.37424659[source]▶
I think that use case still matches the remote cluster use case better as a policy like "We can't use cloud" doesn't mean "we have to use our individual local workstations". This approach really makes sense for the "we have 1-3 people that want to really push this on a budget", beyond that big iron makes more sense. And this still helps with that IMO, it's just one step in getting to there from "only the largest can play".
replies(1): >>37424814 #
2. acdha ◴[07 Sep 23 20:02 UTC] No.37424814[source]▶
Maybe, but that’s potentially slower and definitely much more expensive. A lot of people in those environments can get a $6k workstation a lot faster than a compute cluster which has to be supported, secured, etc.