vibehacker
Discuss
Sam Nguyen
13 hours ago

LM Studio said qwen2.5:14b fit. Activity Monitor disagreed for 40 minutes

#tooling#rants#local-llm

Tried the new qwen2.5:14b-instruct-q4_K_M in LM Studio on the Mac Studio last night. UI said "fits in 64GB" so I loaded it with a 32k context for a RAG eval.

Ten minutes in, fans go nuclear, swap hits 18GB, and my FastAPI worker starts timing out on /embed. Dropped to 8k context and it was fine — still not the "just run 14B locally" story people tweet.

Anyone actually pinning num_ctx + keep_alive in Ollama for evals, or still trusting the green checkmark?

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Riley Cho

    the green checkmark got me too last month. loaded deepseek-coder-v2-lite thinking 36GB was enough, then chrome tabs started dying mid-demo.

    now i just set num_ctx 4096 for anything over 7b and stop pretending the fit estimate knows about my other docker containers. $20/mo cloud embed endpoint still cheaper than replacing this fan.

  • Owen

    yeah the "fits" check only looks at weights, not kv cache. i run ollama with num_ctx 8192 and keep_alive 5m for anything past 7b or the machine starts paging mid-batch.

    also mistral-small on the same box lied about response_format twice last week — model invents keys, zod screams. pin the model tag or you'll chase ghosts.

    • Pixel

      wait so ollama num_ctx is basically a hard cap and lm studio's fit is soft? that explains why my qwen2.5-7b eval was fine and 14b ate the machine even though both "fit".

      also yes on response_format — got {"answer": "...", "confidence": "high-ish"} from mistral-small once. zod hated it. pinning the digest now.

  • Felix Thomas

    We put OLLAMA_NUM_CTX=8192 and keep_alive=0 in the eval compose file after a flaky green run on a shared GPU box. Same story as the LM Studio check — the tool is optimistic about VRAM, the kernel is not.

    If your RAG harness still marks pass when p95 latency jumps 8×, the metric is lying. Pin the ctx in the script, not in a sticky UI toggle.

  • Ash Beacon

    Same class of failure. Green CI on my RAG harness while RAM was thrashing — eval suite still "passed" because timeouts were swallowed in a retry helper.

    I now fail the job if swap delta > 2GB during the run. Ugly metric. Catches the quiet ones though.

More like this

View all