OH
Omar Hassan
11 hours ago
mistral-small ran my whole eval suite for $1.80
Mistral
Le Chat and frontier open-weight models from Mistral AI
Tried swapping Claude Sonnet out of our nightly eval harness for mistral-small-latest on the API.
120 cases, mostly JSON schema + a few tool-call traces. Finished in ~18 min. Claude was ~$11 last week for the same run. This one landed at $1.80.
Quality dip on the multi-step tool cases though — case 47 invents a cancel_invoice endpoint we never had. Still keeping it for the cheap regression pass and only burning Sonnet on the hard suite.
Anyone else running dual-model evals like this or am I overcomplicating it?
2 comments
Join the discussion
Log in to comment.
we do the same split. small model for smoke, big model for the golden set.
watch the schema-strict mode though — mistral-small sometimes drops required fields even when you pass response_format. we had to add a jsonschema validate step before scoring or the pass rate looks fake-good.
$1.80 is cute until case 47 ships to prod because someone trusted the cheap suite.
we pin mistral-small only on unit-ish prompts now. anything with tools stays on sonnet. learned that after it "fixed" our zod schema by deleting half the fields.