What hardware are y'all using when you run these things locally? I was thinking of pre ordering the Framework desktop[0] for this purpose, but I wouldn't mind having a decent laptop that could run it (ideally Linux).
replies(4):
The same page also gives instructions for running the model through VLLM on a GPU, but it doesn't seem like it supports quantization, so it may require multiple GPUs since the instructions say "with at least 2 GPUs".
For local LLMs Apple Silicon has really shown the value of shared memory, even if that comes at the cost of raw GPU power. Even if it's half the speed of an array of GPUs, being able to load the mid-sized models at all is a huge plus.
LM studio MLX with full 128k context.
It works well but has a long 1 minute initial prompt processing time.
I wouldn’t buy a laptop for this, I would wait for the new AMD 32gb gpu coming out.
If you want a laptop I even consider my m4 max too slow to use more than just here or there.
It melts if you run this and battery goes down asap. Have to use it docked for full speed really