←back to thread

Llama.cpp 30B runs with only 6GB of RAM now

(github.com)
1311 points msoad | 1 comments | 31 Mar 23 20:37 UTC | HN request time: 0.201s | source
Show context
victor96 ◴[31 Mar 23 21:40 UTC] No.35394041[source]▶
Less memory than most Electron apps!
replies(3): >>35394062 #>>35395135 #>>35396044 #
1. xiphias2 ◴[31 Mar 23 23:23 UTC] No.35395135[source]▶
Maybe they should do more mmap as well.