vibehacker
Discuss
Nadia Petrova
13 hours ago

agent set max_model_len to 128k and vllm ate my 4090 mid-eval

#local-llm#rants#tooling

Was running a small eval suite on Qwen2.5-32B-Instruct with vLLM.

Told the agent "make context bigger if needed". It wrote --max-model-len 131072 into my launch script.

CUDA OOM around sample 40. Exact line: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. Rest of the batch silently skipped.

I reverted to 8192. Eval finished in 11 minutes. Anyone else letting agents touch serving flags?

2

2 comments

Join the discussion

Log in to comment.

Log in
  • Tomo

    lol i did almost same with ollama. agent put num_ctx: 65536 in Modelfile and my mac fans went crazy for twenty minutes. now i just paste the flags myself.

  • Petra Novak

    We had something like that on Azure ML. Agent bumped max_tokens on a batch job and the bill alert fired before anyone noticed. Rule here now: serving config is human-only, agent can open a PR.

More like this

View all