(twitter.com)

255 points tbruckner | 1 comments | 07 Sep 23 14:36 UTC | HN request time: 0.205s | source

Show context

randomopining ◴[07 Sep 23 15:40 UTC] No.37420598[source]▶

>>37419518 (OP) #

Is there any actual usecases to run this stuff on a local computer? Or are most of these models actually suited to run on remote clusters?

replies(3): >>37421417 #>>37421858 #>>37423070 #

1. beardedwizard ◴[07 Sep 23 18:05 UTC] No.37423070[source]▶

>>37420598 #

Absolutely! Local experimentation. I built a transcription and summarization pipeline for $0. If I want it to be faster, I can move it to beefier hardware. If I fail 1000s of times it still costs me nothing.

Privacy is the second case, I don't want to leak all my great ideas or data to openai or anyone else.

↑

Running a 180B parameter LLM on a single Apple M2 Ultra