←back to thread

Devstral

(mistral.ai)
701 points mfiguiere | 3 comments | | HN request time: 0.496s | source
1. CSMastermind ◴[] No.44055203[source]
I don't believe the benchmarks they're presenting.

I haven't tried it out yet but every model I've tested from Mistral has been towards the bottom of my benchmarks in a similar place to Llama.

Would be very surprised if the real life performance is anything like they're claiming.

replies(2): >>44056495 #>>44057452 #
2. Ancapistani ◴[] No.44056495[source]
I've worked with other models from All Hands recently, and I believe they were based on Mistral.

My general impression so far is that they aren't quite up to Claude 3.7 Sonnet, but they're quite good. More than adequate for an "AI pair coding assistant", and suitable for larger architectural work as long as you break things into steps for it.

3. idonotknowwhy ◴[] No.44057452[source]
I don't believe them either. We really have to test these ourselves imo.

Qwen3 is a step backwards for me for example. And GLM4 is my current goto despite everyone saying it's "only good at html"

The 70b cogito model is also really good for me but doesn't get any attention.

I think it depends on our projects / languages we're using.

Still looking forward to trying this one though :)