vibehacker
Discuss
Tara Nguyen
19 hours ago

Claude invents a Stripe price id and CI still goes green

#rants#agents#tooling

Asked Claude Code to wire a $20/mo Pro plan Friday afternoon. It wrote price_1PxFakeWhatever, mocked Stripe in the test suite, and the PR went green.

Saturday morning three people got charged against a price that doesn't exist in our dashboard. Stripe support was very calm about it. I was not.

Now every agent PR that touches billing needs a human to paste the real price id from the dashboard. Anyone else catching phantom resource ids before merge, or only after the invoice emails?

6

6 comments

Join the discussion

Log in to comment.

Log in
  • Devon

    Mocks are the real bug. We had the same class of failure with a fake prod_xxx Product id — unit tests passed, staging used live Stripe, weekend was chaos.

    I keep a kill-list markdown in every billing repo: no hardcoded price_ / prod_ strings unless they match a checked-in fixtures file we refresh from the dashboard weekly. Ugly. Beats refunds.

  • Moss Harbor

    We put "paste live Stripe ids from dashboard" into the agent acceptance criteria and it still invents them when the prompt mentions "Pro plan".

    What finally stuck: a CI step that greps for price_ / prod_ and fails unless the id is in an allowlist committed from Stripe export. Annoying for PMs. Catching phantom ids before merge is the whole point.

    • Elsa Nowak

      yeah the allowlist from stripe export is the only thing that stuck for us too

      had almost the same friday night: claude code "wired" checkout with price_1ProMonthly, tests mocked the webhook, pr green. monday three EU customers got weird invoices for a plan that never existed in the dashboard

      i now paste the real id into the PR description myself before i even open the agent. if the agent invents one i just close the chat. not worth the saturday

  • Freya Ramirezpro

    Allowlist grep is necessary but not sufficient. We caught Claude writing the inventied id into .env.local and a Helm values overlay — neither is in the allowlist path, both shipped.

    Now the check is: any price_ / prod_ string outside fixtures/stripe.json fails the job, including env files and k8s manifests. PMs hate it. We have zero phantom charges since.

    Also: stop mocking Stripe Checkout in the same suite that asserts invoice amounts. That combo is how you get green CI and Saturday refunds.

    • Evan Liu

      the helm overlay thing is exactly how we got burned. Claude wrote a perfectly plausible price_1ProCAD into values-prod.yaml and our greps only scanned src/.

      we now fail the job if any price_ / prod_ / sk_ / rk_ shows up outside fixtures/stripe.json, including .env*, Helm, and Terraform. PMs hate the false positives. still cheaper than a Saturday refund queue.

      also: never let the agent touch the Stripe secret at all. if it can invent a price id it will invent a key next.

  • Jordan Okoye

    we almost shipped a fake price id last month too. designer me didn't even know Stripe ids looked like that until finance pinged.

    now the rule on our team is dumb and it works: agent can draft the checkout UI, but the price_ line has to be a blank REPLACE_ME until a human pastes from the dashboard. if CI still goes green with REPLACE_ME in it, we failed the gate, not the agent.

More like this

View all