AB
Ash Beacon
10 hours ago
gha restored a stale ollama layer and my eval suite skipped 38 tests
keyed actions/cache on ollama-models-${{ hashFiles('Modelfile') }}. Modelfile only had the model name, not a digest.
runner came up with a half-pulled qwen2.5:7b from last tuesday. fixture hit Connection refused on :11434, whole file flipped to @pytest.mark.skipif, suite green in 41s. found it because the artifacts folder was empty.
anyone pinning ollama pulls by digest in CI, or am i just writing a custom restore step?
5 comments
Join the discussion
Log in to comment.
yeah hashFiles on Modelfile is a trap. we switched to
OLLAMA_MODEL_DIGESTfromollama show --modelfileand fail the job ifcurl -sf localhost:11434/api/tagsdoesn't list that exact digest. skips that turn green are worse than a red suite imo.the digest check is good but
ollama showcan still lie if the blob is truncated. westatthe blob under~/.ollama/models/blobsand require size > 4gb for 7b before pytest even imports.still lost a tuesday to a 200mb ghost layer that looked "present".
same class of bug bit me with a pnpm store cache last month. agent helpfully "fixed" CI by widening the skipif. now i make the fixture assert
len(models) > 0before any test runs — if ollama is dead, the job dies loud in setup.we pin with
ollama pull qwen2.5:7b@sha256:…in a setup step and stick that digest in the cache key. hashFiles on Modelfile alone bit us twice.also set
fail-fast: trueon the matrix so one skipped file doesn't paint the whole check green.repro for the silent skip: fixture catches ConnectionRefused → marks skip. kill that path.
session-scoped fixture that asserts
len(models) > 0with no skip branch. one red setup beats thirty green ghosts.