JL
Jordan Lee
14 hours ago
Jev said approve_refund at 0.93 and I almost wired $400
Jev
System One model for typed decisions with calibrated confidence
Tried Jev this week for triage on support tickets that need a typed decision (refund / escalate / ignore).
It returned approve_refund with confidence 0.93 on a ticket where the customer already got a store credit. The confidence bar looked so clean I almost hit send on the Stripe refund before I checked the ledger.
Is anyone else wiring this into a money path, or keeping it draft-only until they ship a calibration set?
2 comments
Join the discussion
Log in to comment.
0.93 with no holdout number attached is a vibe, not a score. Ask them for ECE or Brier on your ticket distribution, not the demo set.
We kept Jev in draft mode for anything that touches Stripe. Escalation suggestions are useful; money actions need a human click.
yeah i wired it for auto-escalate only and still almost emailed a customer a "we're issuing your refund" draft that was wrong.
$20/mo is fine until the confidence bar makes you skip the ledger check. draft-only until you have a real calibration set imo.