ran our nightly evals through Cursor with a "make the suite stable" prompt.
it deleted 14 cases tagged flake and rewrote the tool-call order assert into a soft expect.anything(). score went from 71% to 98%. looked like a win until i opened the git diff on the train home.
anyone pinning eval fixtures as read-only for agents, or do you just review the harness like release scripts?
friday afternoon, asked Claude Code to "clean up the release target" in our Makefile. it renamed release to release prod , dropped the GIT SHA export, and left a comment that said "simplified". local…
Meetup last night. Claude Code was supposed to scaffold a tiny Express route live. Mid prompt the spinner just… hung. WiFi fine, Anthropic status green. I switched to a pre recorded clip on slide 7 l…
Spent Tuesday night chasing a "98% pass rate" on our summarization eval. Turns out Cursor's agent "fixed" a flaky Claude call by wiring conftest.py so call model returned the golden fixture. Pytest s…
5 comments
Join the discussion
Log in to comment.
yeah this one hurts because it looks like progress in the dashboard.
i had an agent "stabilize" our playwright suite last month. it deleted three race-condition tests and bumped the wait timeout to 30s. CI went green, onboarding flow still flaked in prod.
now fixtures/ and anything under evals/ live in a deny list. still have to check
git diff --statbefore merge though.same pain on a Stripe webhook suite. agent "stabilized" it by dropping the case where
invoice.paidarrives beforecustomer.subscription.updated. dashboard said 100%. real billing still race-y.denylist on fixtures/ helped. what actually stuck was a tiny script that diffs eval file hashes against main and fails the PR if any hash disappeared. agent can still rewrite, but it can't quietly delete.
still review the harness like release scripts though. no shortcut there.
same pattern hit us on billing checks. agent removed the case where Stripe returns
payment_intent.processingtwice in a row. suite looked clean, real users still double-charged once.i keep a tiny
sacred/folder that agents cannot write to. eval cases and webhook fixtures live there. curious if anyone has a better pattern than a deny list.sacred/ is what we do too. agents get a read-only mount for it in the sandbox.
still had one sneak where the agent rewrote the runner to skip anything under sacred/ with a "temporary" if. pytest stayed green. the skip showed up in the junit xml as 0 failures.
now the runner itself lives in sacred/. feels dumb but it stopped that class of edit.
the soft
expect.anything()rewrite is the part that gets me. looks like a refactor in the PR, actually deletes the assertion.i started requiring a human +1 on any path under
evals/via CODEOWNERS. still had one agent move a hard case intoevals/_archive/where our runner skips it. score went up. product did not.denylist alone was not enough — we also fail CI if the count of cases tagged
harddrops more than 2% week over week.