Olmo 3: Charting a path through the model flow to lead open-source AI

(allenai.org)

Show context

thot_experiment ◴[21 Nov 25 08:43 UTC] No.46002543[source]▶

Qwen3-30B-VL is going to be fucking hard to beat as a daily driver, it's so good for the base 80% of tasks I want an AI for, and holy fuck is it fast. 90tok/s on my machine, I pretty much keep it in vram permanently. I think this sort of work is important and I'm really glad it's being done, but in terms of something I want to use every day there's no way a dense model can compete unless it's smart as fuck. Even dumb models like Qwen3-30B get a lot of stuff right and not having to wait is amazing.

replies(3): >>46002752 #>>46005422 #>>46005940 #

1. psychoslave ◴[21 Nov 25 09:19 UTC] No.46002752[source]▶

>>46002543 #

Thanks for the hint. I just tried it on a bright new Mac laptop, and it’s very slow here. But it led me to test qwen2.5:14b and it looks like it can create instant feedback loop.

It can even interact through fluent Esperanto, very nice.

replies(1): >>46002821 #

2. thot_experiment ◴[21 Nov 25 09:29 UTC] No.46002821[source]▶

>>46002752 (TP) #

I'm specifically talking about qwen3-30b-a3b, the MoE model (this also applies to the big one). It's very very fast and pretty good, and speed matters when you're replacing basic google searches and text manipulation.

replies(1): >>46003208 #

3. a96 ◴[21 Nov 25 10:42 UTC] No.46003208[source]▶

>>46002821 #

I'm only superficially familiar with these, but curious. Your comment above mentioned the VL model. Isn't that a different model or is there an a3b with vision? Would it be better to have both if I'd like vision or does the vision model have the same abilities as the text models?

replies(2): >>46003560 #>>46004494 #

4. solarkraft ◴[21 Nov 25 11:40 UTC] No.46003560{3}[source]▶

>>46003208 #

Looks like it: https://ollama.com/library/qwen3-vl:30b-a3b

replies(1): >>46008969 #

5. mark_l_watson ◴[21 Nov 25 13:40 UTC] No.46004494{3}[source]▶

>>46003208 #

This has been my question also: I spend a lot of time experimenting with local models and almost all of my use cases involve text data, but having image processing and understanding would be useful.

How much do I give up (in performance, and running on my 32G M2Pro Mac) using the VL version of a model? For MOE models, hopefully not much.

replies(1): >>46008874 #

6. thot_experiment ◴[21 Nov 25 20:54 UTC] No.46008874{4}[source]▶

>>46004494 #

all the qwen flavors have a VL version and it's a separate tensor stack, just a bit of vram if you want to keep it resident and vision-based queries take longer to process context but generation is still fast asf

i think the model itself is actually "smarter" because they split the thinking and instruct models so both modalities become better in their respective model

i use it almost exclusively to OCR handwritten todo lists into my todo app and i don't think it's missed yet, does a great job of toolcalling everything

7. thot_experiment ◴[21 Nov 25 21:03 UTC] No.46008969{4}[source]▶

>>46003560 #

fwiw on my machine it is 1.5x faster to inference in llama.cpp, these the settings i use for inference for the qwen i just keep in vram permanently

    llama-server --host 0.0.0.0 --model Qwen3-VL-30B-A3B-Instruct-UD-Q4_K_XL.gguf --mmproj qwen3-VL-mmproj-F16.gguf --port 8080 --jinja --temp 0.7 --top-k 20 --top-p 0.8 -ngl 99 -c 65536 --repeat_penalty 1.0 --presence_penalty 1.5

↑