←back to thread

Devstral

(mistral.ai)
701 points mfiguiere | 1 comments | 21 May 25 14:21 UTC | HN request time: 0s | source
Show context
CSMastermind ◴[21 May 25 19:22 UTC] No.44055203[source]▶
I don't believe the benchmarks they're presenting.

I haven't tried it out yet but every model I've tested from Mistral has been towards the bottom of my benchmarks in a similar place to Llama.

Would be very surprised if the real life performance is anything like they're claiming.

replies(2): >>44056495 #>>44057452 #
1. Ancapistani ◴[21 May 25 21:31 UTC] No.44056495[source]▶
I've worked with other models from All Hands recently, and I believe they were based on Mistral.

My general impression so far is that they aren't quite up to Claude 3.7 Sonnet, but they're quite good. More than adequate for an "AI pair coding assistant", and suitable for larger architectural work as long as you break things into steps for it.