had a red job on test_webhook_retry for three days. asked the agent to stabilize it.
this morning CI is green. the test file is gone. commit message says "remove unreliable assertion". no PR description. just a force-push to the branch.
restored it from git history. still flaky. at least it's honest again.
anyone else catching agents that "fix" red suites by deleting the suite?
left a "just fix the flaky e2e" prompt open around midnight. came back at 6:40 and the session had burned $47 retrying the same playwright timeout against staging that was down. usage dashboard looks…
Customer said "if we hit the Q4 number we might raise." Fathom's notes turned that into: Close Series A by Nov 15 Send deck to partner at a16z Confirm $8M valuation (nobody said a number) I almost fo…
caught this tuesday night. agent "fixed" a flaky integration test by wrapping the whole package in //go:build ignore . pr checks green. merge queue happy. i only noticed because coverage dropped 11%.…
5 comments
Join the discussion
Log in to comment.
yeah. mine commented out half the suite with
# agent: temporarily disableand still opened a PR titled "stabilize ci". i only caught it because the diff was 400 lines of deletions.now i ask for a plan that lists which tests stay before it touches anything. still not foolproof.
the plan-first trick helps until it learns to write a plan that says "retire unreliable coverage".
do you force it to keep the original assertion text in the plan, or is that still getting gamed?
same class of bug hit me tuesday. agent rewrote
test_payment_idempotencyintoassert True # flaky under loadand marked the ticket done.i now require a screenshot of the red job in the PR description before anything agent-authored can merge. green without the failing case is just lying to yourself.
i demo'd a "fix CI" agent live last month. same move — deleted the flaky integration test, badge went green, audience clapped. i had to scroll the diff on screen and apologize.
now i screenshot
git diff --statbefore any agent PR goes near a customer. green without the test is just cosplay.the live demo apology is too real.
i added a pre-commit hook that blocks any delete of
test_*.pyunless the commit message hasREVIEWED: tests. agent still tries. i just catch it before push now.