12-prompt eval was green, prod still invented Stripe refund ids
ran a 12-prompt golden set against claude-sonnet and gemini-2.0-flash friday night. both scored 11/12. shipped the flash path for support replies because it was ~$0.004/turn.
monday morning: three tickets where it invented Stripe refund ids that never existed. my eval never had a "look up a real charge then refuse if missing" case — just paraphrase and tone.
anyone else watching tiny green evals greenlight money paths? what do you actually put in the set before you trust a model near billing?
5 comments
Join the discussion
Log in to comment.
11/12 on a set that never exercises tool refusal is not an eval, it is a vibe check.
Before anything touches billing I require at least: missing charge id, expired payment intent, and "user asks for a refund that is already processed". If those three are not in the suite, accuracy is cosplay.
Did your failing case look like a retrieval miss, or did Flash just invent a
re_string when the tool returned empty?this. we shipped a "friendly refund helper" on flash because it was $0.003/turn vs sonnet. monday:
Error: No such refund: re_1SxqFake…in three support tickets.i now wrap the stripe tool: if status != found, the model only gets a fixed enum
NOT_FOUND— no raw json, no empty object it can "helpfully" complete. zod schema on the reply side too. annoying. works.yeah this bit me on a friday ship. eval was "be polite + cite the order id". prod invented
re_3Qx…looking strings because stripe's empty response looked like a soft fail and the model filled the blank.i now force a hard stop: if the tool returns null, the reply template is only "i couldn't find that charge" — no freeform generation after. annoying to write, but cheaper than three angry tickets before coffee.
same hard-stop pattern here, except I also log every null tool result to a slack channel before coffee. berlin 7am, three forged
re_ids in the log, one customer already screenshotting.cheap models stay "helpful" when the tool is quiet. helpful near billing is how you invent refunds. template-only after null is ugly but it stopped the tickets.
Your 11/12 was measuring tone, not tool grounding. I track hallucination rate on money paths separately: last month Flash was 4.2% inventing
re_/ch_shaped strings when retrieve returned empty; Sonnet was 0.9% on the same 80-case set.Minimum suite before I let anything near Stripe: (1) charge exists, (2) charge missing, (3) charge exists but refund already processed, (4) tool timeout. If case 2 is missing, green accuracy is noise.
What did Flash actually return in those three tickets — a full forged id, or a half-plausible
re_prefix with garbage after?