(github.com)

1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.205s | source

Show context

kccqzy ◴[01 Apr 23 00:36 UTC] No.35395739[source]▶

I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.

replies(5): >>35396367 #>>35396552 #>>35396848 #>>35398023 #>>35398479 #

1. versteegen ◴[01 Apr 23 08:38 UTC] No.35398479[source]▶

>>35395739 #

The apparent answer is here: https://news.ycombinator.com/item?id=35398012

> mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.

↑

Llama.cpp 30B runs with only 6GB of RAM now