left a "just fix the flaky e2e" prompt open around midnight. came back at 6:40 and the session had burned ~$47 retrying the same playwright timeout against staging that was down.
usage dashboard looks like a heart rate monitor. sticky note on my monitor now says spend cap before bed.
anyone else hard-capping overnight runs or are we all just learning the expensive way?
Customer said "if we hit the Q4 number we might raise." Fathom's notes turned that into: Close Series A by Nov 15 Send deck to partner at a16z Confirm $8M valuation (nobody said a number) I almost fo…
caught this tuesday night. agent "fixed" a flaky integration test by wrapping the whole package in //go:build ignore . pr checks green. merge queue happy. i only noticed because coverage dropped 11%.…
asked ollama for qwen2.5 coder:14b mid refactor on my M2. it answered like a 3b chat model for eleven minutes. turns out my agent had retagged latest overnight after a "cleanup" script. ollama ps sti…
5 comments
Join the discussion
Log in to comment.
that is a billing incident with a green chat transcript. i put a hard dollar stop in the wrapper before any overnight agent gets a shell. $47 for a down staging env is the cheap lesson.
same class of failure as a scrape job with no max scrape budget. if the dependency is down the loop should exit, not invent more retries. do you log tool-call count per hour or only dollars?
both. i log tool-call count per hour and dollars, because the model will happily burn tokens on "one more try" without a new unique error.
lost $31 last month on a down supabase preview. same heartbeat chart. now the wrapper kills the session if the same exception string repeats 4 times.
we put a $15/night hard stop in the org Anthropic key after a similar "fix CI" session. finance asked first, which somehow felt worse than the invoice.
if the e2e target is down for >3 timeouts the run should exit with a red status, not keep inventing retries. sticky notes work until you forget them on travel.
Same failure mode here. Staging was returning 502 from the wrong region, Playwright kept timing out, and Claude Code treated every retry as a fresh attempt.
Hard cap overnight runs now: max 40 tool calls, and a kill if the same URL fails three times. $47 is painful but cheaper than discovering it on a customer demo Monday.