vibehacker
Discuss
Chris Liu
22 hours ago

mistral-small returned valid JSON with the wrong field names

#local-llm#tooling
Mistral
Le Chat and frontier open-weight models from Mistral AI

running mistral-small:24b on my 3090 via ollama. asked for { "status": "ok" | "error", "latency_ms": number } with their JSON mode on.

got perfectly valid JSON. every time. fields were state, took_ms, and once ok: true with no status at all.

24GB VRAM, ~38 tok/s, looks great in the terminal. then my Zod parse blows up because the schema was a suggestion, not a contract.

anyone pinning a json-repair step after mistral, or did you just switch to a model that actually respects response_format?

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Hao Ward

    We hit the same class of bug in a billing webhook path. Model returned valid JSON, Zod green on "shape", then status was missing and the handler treated it as success.

    What we ship now: AJV against a frozen schema after decode, plus a kill-switch if more than 2% of payloads fail key checks in 5 minutes. Do not let "JSON mode" skip validation — that is how a wrong field becomes a prod write.

    Also: if the model can invent field names, keep payment and auth parsers read-only to the agent. Blast radius is not "retry the call".

  • Owen

    Same boat on a 4090 with mistral-small via ollama. I started keeping a wall-of-shame folder — screenshots where latency_ms became took, duration, even ms once.

    JSON mode staying "valid" is the trap. My FastAPI handler assumed Zod green meant product-correct. It did not.

    Switched models for a week, came back, still hits synonyms. So now it's: decode → rename map for known aliases → strict Zod. Ugly. Works. Anyone found a grammar/constrained decode path that actually sticks on ollama yet?

  • Tess

    yeah we stopped trusting response_format on mistral after it renamed namespace to ns in a helm values patch. ci job pretzel-dry-run stayed green because the yaml still parsed.

    i just run a tiny zod gate + jsonrepair before anything touches the cluster. ugly but cheaper than another mystery rollout.

    • Aria

      wait, pretzel-dry-run stayed green because yaml still parsed?? that's evil.

      i stole your jsonrepair + zod gate idea for a supabase edge function friday night. shipped. saturday morning vitest caught status as ok:true with no status key — green parse, wrong product fields.

      do you pin the repair allowlist in the repo or regenerate it when mistral invents a new synonym?

  • Jonah K

    I kept mistral for the $0 local bill and added a tiny repair queue (brown-line) that only remaps known wrong keys. Anything outside the allowlist goes to a dead letter and pages me.

    Swapping to a cloud model every time this happens cost me more than the sticky-note $20/mo I was trying to avoid. Valid JSON with the wrong field names is still a prod incident — just a quieter one.

More like this

View all