←back to thread

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

(developers.googleblog.com)
602 points emrah | 1 comments | 20 Apr 25 12:22 UTC | HN request time: 0.416s | source
Show context
perching_aix ◴[20 Apr 25 15:19 UTC] No.43744332[source]▶
This is my first time trying to locally host a model - gave both the 12B and 27B QAT models a shot.

I was both impressed and disappointed. Setup was piss easy, and the models are great conversationalists. I have a 12 gig card available and the 12B model ran very nice and swift.

However, they're seemingly terrible at actually assisting with stuff. Tried something very basic: asked for a powershell one liner to get the native blocksize of my disks. Ended up hallucinating fields, then telling me to go off into the deep end, first elevating to admin, then using WMI, then bringing up IOCTL. Pretty unfortunate. Not sure I'll be able to put it to actual meaningful use as a result.

replies(4): >>43744568 #>>43744683 #>>43747309 #>>43748148 #
1. HachiWari8 ◴[20 Apr 25 23:27 UTC] No.43747309[source]▶
I tried the 27B QAT model and it hallucinates like crazy. When I ask it for information about some made up person, restaurant, place name, etc., it never says "I don't know about that" and instead seems eager to just make up details. The larger local models like the older Llama 3.3 70B seem better at this, but are also too big to fit on a 24GB GPU.
replies(1): >>43751992 #