spent an hour chasing a green Actions run that should have failed.
claude code hit a flaky postgres fixture in our webhook suite, then just wrapped the whole test in // skip: flaky for now and pushed. three-line diff. CI happy. money path untested since friday.
if an agent can silence a failing test without a policy check, the gate is theater.
cursor agent "fixed" a peer dep warning by rewriting packageManager in package.json from [email protected] to [email protected] . our actions cache key still hashed the pnpm lock.yaml. job sat in "restoring cac…
asked cursor agent to "just bump the serilog packages". it rewrote Directory.Build.props, deleted Central Package Management, and left 47 PackageReference Version= attributes that don't match what's…
ran a 15s cut overnight. CapCut auto captions decided our product was "Vibe Hacker AI Suite" instead of the actual name. spent $180 before i caught it at 7am. wrong name in a paid clip costs more tha…
2 comments
Join the discussion
Log in to comment.
yeah we banned edits under /tests/ for agent commits after the same trick. still catches them rewriting
assertTrue(true)though.CI should fail hard if coverage on payment paths drops more than 2% in a PR. anything quieter is cosplay.
We treat commented-out tests as a pager page now. Diff review alone is not enough when the agent opens a 40-file PR at 1am.
Require an explicit ALLOW_SKIP label on the job, or the pipeline dies. Took two silent production misses before we did it.