←back to thread

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

(developers.googleblog.com)
602 points emrah | 2 comments | 20 Apr 25 12:22 UTC | HN request time: 0.433s | source
Show context
perching_aix ◴[20 Apr 25 15:19 UTC] No.43744332[source]▶
This is my first time trying to locally host a model - gave both the 12B and 27B QAT models a shot.

I was both impressed and disappointed. Setup was piss easy, and the models are great conversationalists. I have a 12 gig card available and the 12B model ran very nice and swift.

However, they're seemingly terrible at actually assisting with stuff. Tried something very basic: asked for a powershell one liner to get the native blocksize of my disks. Ended up hallucinating fields, then telling me to go off into the deep end, first elevating to admin, then using WMI, then bringing up IOCTL. Pretty unfortunate. Not sure I'll be able to put it to actual meaningful use as a result.

replies(4): >>43744568 #>>43744683 #>>43747309 #>>43748148 #
1. parched99 ◴[20 Apr 25 15:52 UTC] No.43744568[source]▶
I think Powershell is a bad test. I've noticed all local models have trouble providing accurate responses to Powershell-related prompts. Strangely, even Microsoft's model, Phi 4, is bad at answering these questions without careful prompting. Though, MS can't even provide accurate PS docs.

My best guess is that there's not enough discussion/development related to Powershell in training data.

replies(1): >>43746262 #
2. fragmede ◴[20 Apr 25 20:18 UTC] No.43746262[source]▶
Which, like, you'd think Microsoft has an entire team there who's purpose would be to generate good PowerShell for it to train on.