FR
Freya Ramirezpro
14 hours ago
System prompt said never invent APIs. It still shipped fake Stripe webhooks
Spent Thursday rewriting the system prompt: "never invent APIs, only use routes listed in openapi.json."
Claude still invented POST /v1/webhooks/stripe/retry with a neat TypeScript client and three helper functions. Vitest even passed because the mock returned 200.
Caught it when staging returned 404 and Stripe dashboard stayed empty. Anyone running a hard allowlist of paths the agent can touch, or do you just diff harder?
5 comments
Join the discussion
Log in to comment.
Allowlist is the only thing that stuck for me.
I pipe the agent's proposed paths through a Go map keyed off the real openapi.json. Anything not in the map gets a hard error before Vitest even runs. Prompt yelling "don't invent" is theater — the model treats it like a suggestion.
lol this happened to me friday. agent invented
POST /billing/stripe/replayand the mock just shrugged.now i have a pre-commit hook that greps for routes not in openapi. ugly. works. staging 404s still haunt me though
This is why I force the prototype to call a typed client generated from the real OpenAPI before any PRD goes out.
If the agent can't import the function, it doesn't exist. Diff review still catches inventiveness, but the compile step is the kill criteria — no green CI on hallucinated paths.
wait — typed client from OpenAPI as the kill criteria is smart. do you regen the client on every openapi change in CI, or is it a checked-in artifact?
we tried
stripe listen+ a fixture recorder and the agent still "helpfully" added a retry endpoint that only existed in the mock. green CI, empty dashboard. classic.same energy in tutoring agents. mine invented
GET /lessons/{id}/hints-v2last week and the mock was "helpful" so tests were green.real student hit 404 mid-homework. now i keep a yaml allowlist of routes and fail the agent loop if it emits anything else. messy but kids notice faster than your staging alerts.