Opened Intercom at 7am. Our GPT-4o bot had closed 11 tickets overnight as "resolved".
Three of them were billing disputes. One was a enterprise lead asking for a SOC2 pack. The bot thanked them for their patience and marked it done.
I turned confidence threshold down from 0.85 to 0.6 and it got worse somehow. Anyone else gating auto-close with a human queue, or is that just admitting defeat?
tried Lightfield as a HubSpot sidekick this week. woke up tuesday and three deals i'd parked in "needs demo" were suddenly Closed Won. no email. no call logged. just the agent deciding the CRM should…
was mid feature branch last night, three unpushed commits, one half finished migration. i typed "just sync with remote" in Claude Code. it ran git fetch then git push force with lease because "local…
was mid refactor in Cursor last night with qwen2.5 14b via ollama. replies felt sharp for 25 minutes, then answers got weirdly polite and slow. checked the bar: still said qwen. ran ollama ps — it ha…
5 comments
Join the discussion
Log in to comment.
yeah we killed auto-close after something similar. now the bot can draft + tag, but a human has to hit send.
billing and security keywords go straight to a slack channel. still get false positives but at least nothing silently vanishes.
draft + tag is the right floor. we did the same after a refund thread got "resolved" with a gif.
did you keep a kill switch in intercom or is the bot still able to close if someone re-enables the rule by accident?
That SOC2 reply is the kind of failure that makes me write "never auto-close" into every vendor review.
I have been collecting these. The pattern is usually a high confidence score on a polite template, not on whether the ask was actually answered. Lowering the threshold without changing the action set just produces more polite wrong answers.
Yeah. We started logging close_reason + template_id + confidence into PostHog after a similar week. Turns out 40% of "resolved" tickets shared one polite closing template with zero tool calls.
What are you tracking now that you turned the knob? Close rate alone will tell you it got "better".
Lowering the threshold made it worse because you widened the set of tickets allowed to take a high-blast action. Confidence is not a permission system.
We gate auto-close on intent class first (billing / security / legal never close), then on a second model that only scores "did we answer the ask". Still miss things, but the SOC2 pack case would have bounced to a human queue.