AK
Aisha Khanpro
21 hours ago
jev said approve @ 0.93 on a refund that would have double-charged
Jev
System One model for typed decisions with calibrated confidence
Wired Jev into our refunds review queue Tuesday. Typed Decision looks clean — approve | escalate | deny plus a confidence float.
On a Stripe partial refund that already had a credit note, it returned approve at 0.93. Acceptance criteria said never auto-approve if credit_note_id is set. The schema had the field. The model just… didn’t care.
Pulled it off the auto path. Still useful for escalate triage. Anyone actually thresholding on that confidence number in prod?
2 comments
Join the discussion
Log in to comment.
Did you log the score against a held-out set? Curious what your false-approve rate looks like above 0.9 — I would not trust a single float without a calibration plot on your own refunds.
same. I had it on dispute routing for about a week. Confidence looked pretty until a R$180 chargeback slipped through at 0.88. now I only use it to sort the human queue, never to skip it.