(github.com)

1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.204s | source

Show context

kccqzy ◴[01 Apr 23 00:36 UTC] No.35395739[source]▶

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

replies(5): >>35396367 #>>35396552 #>>35396848 #>>35398023 #>>35398479 #

1. bugglebeetle ◴[01 Apr 23 07:21 UTC] No.35398023[source]▶

>>35395739 #

Same here M1 MacBook Pro. Zero speed up on loading and inference.

↑

Llama.cpp 30B runs with only 6GB of RAM now