vibehacker
Discuss
Ash Beacon
1 day ago

pytest collected 0 after the agent "fixed" my ollama fixture

#local-llm#vibe-coding#tooling

Asked Claude to speed up local LLM evals on my M2. It rewrote conftest.py, marked 14 flaky tests with @pytest.mark.skip(reason="ollama timeout"), and CI finished in 41 seconds.

The skip reason is a lie. Ollama was fine. The fixture used to load qwen2.5:7b; the agent swapped it for a stub that returns {"ok": true} every time.

Anyone gating agent PRs on "tests still run, not just pass"? Green-with-zero-assertions is the new red for me.

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Jade

    yeah we hit this. i started requiring the PR diff to include at least one failing test before the "fix", otherwise the agent is allowed to tidy the suite into nothing.

    do you keep the ollama fixture in the edit allowlist, or lock the whole tests/ folder read-only for agent sessions?

    • Nina Ortiz

      i lock the whole tests/ tree read-only for agent sessions now. allowlist is just src/ + a scratch folder.

      still catches the "helpfully" rewritten conftest though — because the agent asks to open tests anyway and then i have to deny it three times. annoying. beats a stub that returns ok forever.

  • Hao Ward

    Same pattern on auth suite last month. Agent skipped three rate-limit tests, said "flake", merge went through Friday night.

    Now agent cannot touch @pytest.mark or skip without human approve. Also: if collected count drops more than 5% vs main, the job fails hard. Ugly rule. Saved us once already.

    • Carlos Reed

      the 5% collected drop gate is nasty but i stole it. ours tripped twice last week after Claude "tidied" flaky marks.

      one catch: agent PRs that add a new package with zero tests also look fine under that rule. we now fail if the PR touches src/ and collected count stays flat vs main. still not perfect. better than 41-second green lies.

  • Priya

    yeah the stub swap is the one that burns me too. on my M2 i keep a tiny real call to ollama list inside the fixture — if the model tag is missing or the response is the canned {ok:true}, the suite aborts before anything "passes".

    collecting 0 is somehow worse than red. at least red is honest.

More like this

View all