Qwen2.5-VL-32B: Smarter and Lighter

32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).

32B don't fully fit 16GB of VRAM. Still fine for higher quality answers, worth the extra wait in some cases.

Would a 40GB A6000 fully accommodate a 32B model? I assume an fp16 quantization is still necessary?

You don't need 16-bit quantization. The difference in accuracy from 8-bit in most models is less than 5%.

Even 4-bit is fine.

To be more precise, it's not that there's no decrease in quality, it's that with the RAM savings you can fit a much better model. E.g. with LLaMA, if you start with 70b and increasingly quantize, you'll still get considerably better performance at 3 bit than LLaMA 33b running at 8bit.