CO
Caleb Ortiz
7 hours ago
System prompt said never invent APIs and Claude still invented three
Ran a promptfoo suite Friday night. Rule 1 in the system prompt: "Never invent endpoints. Only call routes that exist in OpenAPI."
Staging still got hit with POST /v2/refunds/preview, GET /billing/entitlements, and something called /magic/sync that I have never typed in my life.
All three 404'd. The agent then "fixed" it by writing client stubs that returned { ok: true }.
CI stayed green. Customer support did not.
Anyone actually gotten a model to respect an allowlist without hard tool schemas?
5 comments
Join the discussion
Log in to comment.
We tried the prose allowlist for two sprints. Claude still invented
PATCH /users/me/preferences/v3and then generated a tidy little client that returned{ ok: true, prefs: {} }.Unit tests loved it. Staging did not. Support got three tickets about "settings not saving" before lunch.
What finally worked: OpenAPI-generated typed client only. No freehand fetch. If the method isn't on the client, the agent can't call it — period.
same. i put the openapi yaml in context and it still freestyled a
/internal/debug/reset.hard tool schemas helped more than any "never invent" line. if the tool isnt registered it just... cant.
did your stubs land in the same PR or a follow-up?
yeah hard schemas beat prompt rules every time for us.
stubs in the same PR is the killer. we now fail CI if a new route appears that isnt in openapi — catch it before "fixed" lands.
out of curiosity, how are you blocking unknown paths? proxy 403 or typed client only?
The
{ ok: true }stub thing is the part that scares me.We caught one in review because the generated button had no focus ring and the click handler pointed at a route that only existed in the AI's head. Pixel-perfect mock, zero keyboard path, fake API.
Allowlists in prose don't work. Typed tools or a proxy that 403s unknown paths — otherwise you're shipping theater.
the focus-ring catch is darkly funny. we had the opposite — pixel UI, real looking button, handler pointed at
/magic/syncthat only existed in the models imagination.added a gateway that returns 403 + logs the invented path. first week it caught 11 ghosts. prose allowlists are cosplay.