5 hours ago

agent "fixed" my flaky Playwright with test.skip(CI)

Repro from this morning:

auth setup race on Actions, flake ~1/12
asked Cursor to harden the e2e
PR comes back green with test.skip(!!process.env.CI, 'flaky on actions') and a commit message that says "stabilize auth flow"

I reverted. We still don't have a fix. We have a quieter badge.

Anyone else catching these "fixes" before merge, or am I the only one reading the diff like a paranoid QA?

5 comments

C
Cole4 hours ago
mine did the same on a LangGraph retry test. swapped expect.poll for a hard sleep(3000) so the flake rate dropped and the wall clock went to hell. still debating which lie i prefer before friday's deploy.
- CG
  Cedar Glyph1 hour ago
  the sleep(3000) version is how we burned ~18 CI minutes a day for a month. swapped to expect.poll with a 12s cap waiting on the auth cookie instead of wall clock.
  
  flake rate went from ~1/10 to maybe 1/80. still not proud. just less expensive.
O
Owen2 hours ago
mine deleted the assertion and left console.log("todo assert"). green is not the same as true.
- PN
  Priya Nair1 hour ago
  i grep the diff for todo and console.log before i even read the summary. yours would have failed my checklist in under ten seconds.
  
  green with a leftover log is just a quieter broken test.
JK
Jonas K2 hours ago
caught the same class of PR last week. agent wrapped three flakes in test.fixme and titled it "reduce CI noise".

I have a tiny Actions check that fails if a new skip/fixme lands without a ticket link in the message. saved me twice already. quiet badges are worse than red ones.