Prompt A/B won activation and also set support on fire
Shipped a system-prompt tweak Friday afternoon. PostHog said activation +4% over the weekend.
Support also got 19 tickets about the bot "sounding rude" and one user who got a refund path that wasn't supposed to exist anymore. We use Claude 3.5 / Sonnet via the API, $20/mo plan on our side for the staging key… not the point.
Anyone else A/B prompts like pricing experiments, or am I the only one whose "winner" also lights the help desk on fire?
5 comments
Join the discussion
Log in to comment.
Did you hold out a fixed eval set, or was the A/B just live traffic?
We've had the same trap — metric moves, tone drifts, and nobody noticed until Slack lit up. Without ~50 labeled failures frozen before the change, I wouldn't trust a +4% over a weekend.
Same. We keep ~40 "should never say X" fixtures in CI now — rude tone, phantom refunds, inventing UI.
Activation A/B without that suite is how you wake up to 19 tickets and a Sonnet bill that still says $20/mo like nothing happened.
We treat ticket volume as a kill switch now. Last time activation jumped 6% the Intercom queue spiked the same day — model started offering free months it couldn't honor.
If support isn't a secondary metric in the A/B, you're just optimizing for a chart.
yeah i once "won" an a/b by telling the model to be more direct. conversion ticked up. then it started answering billing questions with "just cancel lol"
now i keep the prompt in git and a stupid checklist: tone, refund path, never invent buttons. still ship friday sometimes though.
the checklist thing is underrated. i literally keep a sticky: "does it invent buttons? refunds? cancel paths?"
still shipped a friday prompt once that told users to email founder@… which was my personal gmail. activation was fine. sleep was not.