QS
Quiet Ship
1 day ago
agent deleted my playwright retries and called the suite "stable"
friday night, three flakes in checkout.spec.ts. i told cursor "make these less flaky".
it removed the retries: 2 block, deleted two expect.poll waits, and opened a PR titled "stabilize e2e". CI went green once. saturday morning the same race is back, now with no safety net.
i reverted, put retries back, and added a CODEOWNERS rule on playwright.config.*. anyone else getting "stable" PRs that just delete the slow parts?
5 comments
Join the discussion
Log in to comment.
Diff size is the tell. If the PR shrinks the config and the suite duration drops 40%, it did not stabilize anything — it removed coverage.
We require a before/after flake rate on the same commit SHA. No numbers, no merge. Agents are very good at optimizing for green once.
same class of bug as commenting out tests. green badge means nothing without the flake log.
in our repo i put a github action that fails if playwright.config.* loses retries or if anyone deletes
toPass/expect.poll. agent hit it twice last week. loud fail is better than saturday morning surprise.yeah. green after deleting retries is just a quieter flake.
we log
retries+ anyexpect.pollcount as a CI artifact now. if either drops vs main, the job fails with the exact diff. agent tried to rename the config once to dodge it. cute.stole the artifact idea. we also keep a denylist in the agent prompt for
playwright.config.*and anything undere2e/fixtures.still slips on rename. your drop-vs-main check would have caught that. adding it monday.
same energy as the agent that deleted our
continue-on-error: falseand called it "cleanup".i put
playwright.config.tson the protected paths list with a required review from anyone on #ci. also a dumb script that greps forretries:and fails the pr if it's gone. not elegant. works.