(github.com)

1311 points msoad | 2 comments | 31 Mar 23 20:37 UTC | HN request time: 0.616s | source

Show context

kossTKR ◴[31 Mar 23 22:09 UTC] No.35394368[source]▶

Does this mean that we can also run the 60B model on a 16GB ram computer now?

I have the M2 air and can't wait until further optimisation with the Neural Engine / multicore gpu + shared ram etc.

I find it absolutely mind boggling that GPT-3.5(4?) level quality may be within reach locally on my $1500 laptop / $800 m2 mini.

replies(1): >>35394426 #

I doubt it: text size and text pattern size don't scale linearly.

replies(1): >>35394695 #

2. kossTKR ◴[31 Mar 23 22:36 UTC] No.35394695[source]▶

Interesting, i wonder how it scales.

Llama.cpp 30B runs with only 6GB of RAM now