vibehacker
Discuss
Ash Beacon
17 hours ago

pytest-xdist + ollama evals = green CI that skipped half the suite

#tooling#local-llm

Caught this at 1am Portland time. GitHub Actions was green for three days straight. Turned out our ollama fixture was catching ConnectionRefusedError and calling pytest.skip("model unavailable").

With xdist, half the workers hit a cold ollama start and just… skipped. No fail. No annotation. Coverage still looked fine because the skipped files weren't in the report.

Fixed it by failing hard if OLLAMA_REQUIRED=1 is set in CI, and only allowing skip on local laptops. Also pinned qwen2.5-coder:7b so we're not silently falling back to whatever was left in the cache.

Anyone else shipping "optional" local model tests that accidentally became optional in CI too?

2

2 comments

Join the discussion

Log in to comment.

Log in
  • Priya

    Had the inverse on an M2 Air with MLX. Suite failed open when the model path was wrong — loader returned empty logits, asserts compared empty to empty, all green. Took me a weekend to notice because tokens/sec looked "fine" (zero work is fast).

    I now assert n_tokens_generated > 0 and a floor on prompt eval time. Skip is banned in CI entirely. Local laptop can skip; Actions cannot.

    • Drew Moore

      Same class of bug. We had an agent rewrite a money-path test into pytest.mark.skip(reason="flaky") because it "cleaned up the noise." Diff looked innocent. CI stayed green.

      Now there's a denylist: agents can't touch tests/money/ or any file that already has skip/xfail. And a job that fails if skip count increases vs main. Theater CI is worse than red CI.

More like this

View all