KW
Kenji Watanabepro
2 days ago
Modelfile said num_ctx 32768 but ollama still truncated at 8192
spent tuesday night thinking my RAG eval was broken.
i set PARAMETER num_ctx 32768 in the Modelfile for qwen2.5:14b-instruct-q4_K_M. ollama show printed 32768. then every long prompt still cut around 8k tokens — no error, just quiet chop.
turns out the running container was still the old tag from last week. i forgot ollama create after edit. Activity Monitor looked fine so i blamed the embedding model for two hours. sorry for english.
do people pin a digest / hash for local tags, or just re-create every time and pray?
5 comments
Join the discussion
Log in to comment.
yeah ollama show lying about the live process is a classic. i now
curl localhost:11434/api/psbefore every eval night. if the digest does not match the Modelfile hash i wrote down in a sticky note, i kill it.Mac Studio RAM looks "fine" while the model is half-loaded and silently capped. 14b + tight context still beats a ghost 32k you never actually got.
the sticky note method is too honest. i wrote a 3-line shell that diffs
ollama show --modelfileagainst the file on disk before any eval. caught the stale tag twice this week on a mac mini.14b + real 8k still beats 32k on paper. hate that.
same energy as when cursor said the mcp was connected and tools[] was empty. the dashboard lied, the runtime did not.
i keep a one-liner in my friday checklist now: recreate the ollama tag, then hit /api/ps, then run one 12k prompt that MUST fail if ctx is still 8k. if it succeeds quietly you're still on the old box.
spent a whole evening once chasing "bad retrieval" that was just truncated context. never again.
the 12k must-fail prompt is the right move. i keep a fixture with a 10k token dump of nonsense + a needle at the end. if the model can't echo the needle, context is fake.
also recreate the tag with an explicit name like
qwen2.5-14b-ctx32k-v3so last week's ghost can't win.i pin the digest in a Makefile target now.
ollama createthen curl /api/ps and fail the job if context_length is still 8192.lost a whole thursday to silent truncation on mlx too. the show command is for humans, not for CI.