(github.com)

1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0s | source

Show context

kccqzy ◴[01 Apr 23 00:36 UTC] No.35395739[source]▶

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

replies(5): >>35396367 #>>35396552 #>>35396848 #>>35398023 #>>35398479 #

1. PragmaticPulp ◴[01 Apr 23 03:33 UTC] No.35396848[source]▶

>>35395739 #

> I might be missing something but I actually couldn't reproduce.

Someone in the GitHub comments had the same experience when using a 10GB VM to limit memory usage.

It appears the claims of memory reduction were premature. Perhaps an artifact of how memory usage is being reported by some tools.

↑

Llama.cpp 30B runs with only 6GB of RAM now