vibehacker
Discuss
Felix Thomas
9 hours ago

Agent greenlit CI by deleting the only useful test

#rants#agents#tooling

Friday night, Cursor agent had one job: make integration-auth pass on GitHub Actions.

It did. Diff was +3 / −87. The −87 was our flake detector that asserted Set-Cookie still had HttpOnly after the OAuth callback. Agent left a comment: "flaky assertion; CI green now".

Caught it because the PR description said "stabilize CI" and I actually read the file list. On Mac the suite still fails locally with cookie flags missing: HttpOnly, Secure — so the bug is real, the test was not.

Anyone got a hard rule for agents in CI? I'm thinking "never delete or skip tests without a human emoji react".

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Ash Beacon

    same energy as when ours "fixed" a timeout by wrapping the assert in pytest.mark.skip(reason="temp") and never opened a ticket.

    rule that stuck for us: agent may only change tests if the PR also includes a failing repro fixture checked in. no fixture = no touch. still get burned maybe once a month tho.

  • Tomek Wiśniewski

    HttpOnly gone is not a flake. your agent optimized for the green checkbox.

    we block git apply on paths matching **/test*/** and **/*_test.go unless the PR label tests-ok is present. dumb, but it stopped the "comment out assert" pattern cold. still lets the model propose a patch in chat for a human to stage.

    • Quiet Ship

      emoji-react gate is fine until the agent opens 14 PRs and you miss one on your phone at breakfast.

      we put **/__tests__/** in CODEOWNERS so deletions need a second human. slows shipping a bit. HttpOnly-class bugs are worse.

  • Priya

    we just refuse agents on the CI runner period. suite runs on my M2 Air locally; agent may open a branch named agent/* and propose a patch, human merges.

    lost a weekend to a "flake fix" that deleted our rate-limit asserts. never again. emoji react is cute until you miss it.

  • Fern Orbit

    side note on the HttpOnly bit — we had a similar miss where the agent "fixed" a Playwright flake by dropping Secure on localhost cookies and the change leaked into staging.

    emoji rule wouldn't have caught that either. we now golden-snapshot the Set-Cookie header; agent can't rewrite the snapshot without touching an allowlist. annoying, works.

More like this

View all