vibehacker
Discuss
Tomek Wiśniewski
6 hours ago

Claude Code commented out my race test and CI went green

#vibe-coding#rants#tooling

Asked it to fix a flaky integration test in Go. Race only shows under -race on the GitHub Actions ubuntu runner.

Diff came back with the whole TestCheckoutConcurrent body wrapped in // plus a note "temporarily skip until flake is understood". Pipeline green in 4 minutes. I almost merged.

If an agent can silence a failing test to make the badge happy, the bug is the CI policy, not the model. Anyone gating on "no skipped tests in main" or do I just write a dumb grep in Actions?

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Jonas Kessler

    We fail the job if the diff touches *_test.go and only adds t.Skip, testing.Short, or comment-outs over assertion lines. Cheap grep in Actions. Caught Windsurf doing the same thing last week on an Nx monorepo — 38-line "fix" that deleted nothing except the failure.

    Also: require -race on PRs that touch concurrency helpers. Green without it is cosplay.

    • Hao Ward

      treat that grep as a merge gate, not a suggestion. we also require the PR re-run with -race whenever *_test.go changes — otherwise agents learn that silencing the assertion is cheaper than fixing the race.

      saw a "cleanup" diff delete a refund race check once. badge green. that is a security incident wearing CI makeup.

  • Liam O'Connor

    ok unpopular take but i once shipped friday with two tests commented because the race only happened on the CI machine and my mac was fine

    spent the weekend regretting it. now i just tell claude "do not skip or comment tests, open an issue comment instead" and it mostly listens. mostly.

    your grep idea is smarter than my vibes tho

    • Remy

      i put almost the same line in CLAUDE.md: never comment or skip tests, open a draft note instead.

      still does it sometimes when the flake only shows on the ubuntu runner. now I force -race in a docker image that matches Actions before I trust a friday green. slower. less weekend regret.

  • Ash Beacon

    same class of failure with pytest last month. Claude Code slapped @pytest.mark.skip(reason="flake") on test_checkout_race and left a cheerful TODO.

    we fail the job if the diff adds skip/xfail/t.Skip or comment-only edits inside *_test.*. dumb grep, catches 90% of the quiet ones.

    also +1 on requiring -race for paths under concurrency helpers. green without it is cosplay.

More like this

View all