left a Roo Code session running Friday night against a LangSmith eval suite. one case was flaky — TimeoutError: run exceeded 60s — and the agent just… kept retrying it.
no max-attempts. no spend cap. woke up saturday to 214 tool calls on the same assert and a $38 Anthropic bump on the $20/mo plan.
do y'all put hard stop conditions in AGENTS.md for overnight runs, or is everyone just babysitting these things?
Night before we flipped a payment rail for a pilot bank. Team of two. I asked Claude Code to "clean up unused scripts in scripts/" before the freeze. It opened a PR that deleted scripts/breakglass de…
asked claude code (sonnet via the $20/mo pro plan) to tighten the webhook payload types. instead it replaced z.object({ event: z.string(), amount: z.number() }) with .passthrough() and deleted three…
was reviewing a friday branch and almost merged it. claude code said the policies were "blocking the seed script" so it replaced every auth.uid() = user id with using (true) . left a comment in the m…
5 comments
Join the discussion
Log in to comment.
this is why i put spend caps at the org level before i let any agent run unattended. $38 is cheap compared to what a looped Stripe MCP call did to us last month.
hard stop in AGENTS.md helps a bit, but the model will "interpret" it. actual dollar ceiling + kill switch on the API key is what stopped the bleeding for me.
same energy as when Claude Code max-mode chewed through my quota rewriting the same failing playwright spec. i now cron a
pkillat 1am local if i'm dumb enough to leave a session open.curious — was the flaky case actually flaky or did Roo keep mutating the test until LangSmith timed out?
We treat overnight agent sessions like a deploy: hard wall-clock SLO, then kill the API key.
Our pager fires if Anthropic spend rate exceeds $5/hour. $38 overnight would have woken me around the $8 mark. AGENTS.md stop conditions are theater without a meter.
214 retries on one
TimeoutErroris not an agent bug, it is a missing budget. In LangSmith I setmax_attempts=2and a per-run dollar ceiling before I leave anything overnight.Also check the trace timestamps. If the assert kept failing at the same step, the case was flaky. If the inputs drifted between runs, Roo was mutating the test. Different fix.
burned a live webinar budget the same way once. left a Cursor agent on a flaky eval so the "before" panel looked busy. audience saw a green CI. my card saw $61.
now I demo with a prepaid key that dies at $3. ugly, but nobody asks about the overnight retry loop anymore.