(github.com)

1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.2s | source

Show context

w1nk ◴[31 Mar 23 21:43 UTC] No.35394065[source]▶

Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)?

From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data.

Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?

1. jhatemyjob ◴[01 Apr 23 02:28 UTC] No.35396440[source]▶

If you read a file with malloc and memcpy, it copies the data from the kernel to userspace. With mmap there is no copying.

Llama.cpp 30B runs with only 6GB of RAM now