←back to thread

DeepSeek-v3.1

(api-docs.deepseek.com)
776 points wertyk | 1 comments | | HN request time: 0.215s | source
Show context
danielhanchen ◴[] No.44978800[source]
For local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow.

./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU"

More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1

replies(6): >>44979837 #>>44980406 #>>44981373 #>>44982860 #>>44984274 #>>44987809 #
pshirshov ◴[] No.44979837[source]
By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?
replies(2): >>44980068 #>>44981691 #
exe34 ◴[] No.44981691[source]
hey fellow crazy person! slight tangent: one thing that helps keep me grounded with "LLMs are doing much more than regurgitation" is watching them try to get things to work on nixos - and hitting every rake on the way to hell!

nixos is such a great way to expose code doing things it shouldn't be doing.

replies(2): >>44981751 #>>44987224 #
pshirshov ◴[] No.44987224[source]
In my experience LLMs can do Nix very well, even the models I run locally. I just instruct them to pull dependencies through flake.nix and use direnv to run stuff.
replies(1): >>44989028 #
1. exe34 ◴[] No.44989028[source]
oh yes they do nix very well, but I asked cursor to set up a firecracker vm with networking for exposing a port on the host, and use conda inside to install a certain version of python with some libraries. I asked for a firecracker-vm.nix, a build.sh, a run.sh and a close.sh. it kept trying to run code inside its own fhs-env, which would run, and then when I tried it outside of the fhs, it would fail. I'd paste in the errors and it would without fail say oh let's try the proper nix version of python - which I explicitly didn't want, because I wanted to run conda versions on other machines. I tried to guide it through conda-shell but didn't get very far. in the end I ended up using docker instead, which it did set up without fail.

but when it was failing on my original idea, it kept trying dumb things that weren't really even nix after a while.