Apart from using a Mac, what can you use for inference with reasonable performance? Is a Mac the only realistic option at the moment?
This one should just about fit on a box with an RTX 4090 and 64GB RAM (which is what I've got) at q4. Don't know what the performance will be yet. I'm hoping for an unsloth dynamic quant to get the most out of it.
Whats important is VRAM, not system RAM. The 4090 has 16gb of VRAM so you'll be limited to smaller models at decent speeds. Of course, you can run models from system memory, but your tokens/second will be orders of magnitude slower.
ARM Macs are the exception since they have unified memory, allowing high bandwidth between the GPU and the system's RAM.