Gemma 3 QAT Models: Bringing AI to Consumer GPUs

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern).

The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they knew the right tools.

Ollama is definitely not for production loads but vLLm is.

I did not know this, so thank you. I read a blogpost a while back that encouraged using Ollama and never mention vLLM. Do you recommend reading any particular resource?

The significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though.

It is important to know about both to decide between the two for your use case though.

Why is sillytavern the only LLM frontend which matters?

instead of ranting, maybe explain how to make a qat q4 work with images in vllm, afaik it is not yet possible

Last I looked vLLM didn't work on a Mac.

Somebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?

FWIW I'm getting 29 TPS on Ollama on my 7900 XTX with the 27b qat. You can't really compare inference engine to inference engine without keeping the hardware and model fixed.

Unfortunately Ollama and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet.

https://github.com/vllm-project/vllm/issues/16856

I tried sillytavern a few weeks ago... wow, that is an "interesting" UI! I blundered around for a while, couldn't figure out how to do anything useful... and then installed LM Studio instead.

I personally thought the lorebook feature was quite neat and then quickly gave up on it because I couldn't get it to trigger, ever.

Whatever those keyword things are, they certainly don't seem to be doing any form of RAG.

Afaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-infere...

Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.

It is much faster on single prompts than ollama. 3X is not unheard of

It supports more sampler and other settings than anyone else.

What command is it? Because that was not at all my experience.

Vllm serve… huggingface gives run instructions for every model with vllm on their website.

How do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.

From the HN guidelines: https://news.ycombinator.com/newsguidelines.html

> Be kind. Don't be snarky.

> Please don't post shallow dismissals, especially of other people's work.

In my opinion, your comment is not in line with the guidelines. Especially the part about sillytavern being the only LLM frontend that matters. Telling the devs of any LLM frontend except sillytavern that their app doesn't matter seems exactly like a shallow dismissal of other people's work to me.

Had to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.