spent friday night trying to rename a package with mlx-community/Qwen2.5-32B-Instruct-4bit on an M2 Max 64GB.
got through the import graph, wrote a half-finished pyproject.toml, then metal panic and the chat history was gone. cloud sonnet finished the same rename in like 4 mins and i feel dirty.
anyone actually shipping local models for refactors that touch more than one file, or is that still a party trick?
Tried the new qwen2.5:14b instruct q4 K M in LM Studio on the Mac Studio last night. UI said "fits in 64GB" so I loaded it with a 32k context for a RAG eval. Ten minutes in, fans go nuclear, swap hit…
Sorry English. I pipe llama 3.3 70b through Groq for agent loops because latency look nice on demos. Yesterday: 900 tok/s for first 4 calls, then hard 429 for 40 minutes while I still had half a Type…
running mistral small:24b on my 3090 via ollama. asked for { "status": "ok" | "error", "latency ms": number } with their JSON mode on. got perfectly valid JSON. every time. fields were state , took m…
5 comments
Join the discussion
Log in to comment.
The half-finished pyproject is the part that hurts. I keep a local model for UI copy and component stubs, but anything that rewrites imports goes to a hosted model with a git checkpoint first. Learned that after it ate my tokens.css once.
did you pin the mlx version? i had the same segfault after a brew upgrade last week — rolled back to 0.22.x and the rename at least completed. still slower than Claude for multi-file though. rough evals on my side: local wins on latency for single-file, loses hard on tool use.
sorry english. for mlx i pin num_ctx to 8192 even if model card says 32k — default window eats RAM then metal dies on tool calls.
14B + tight ctx more stable for multi-file than half-loaded 32B. cloud still wins when you need reliable tool use, yes.
32B Q4 on a 64GB Mac is basically asking Metal to lie about headroom. I run Qwen2.5-14B-Instruct-Q5_K_M for renames and keep 32B for chat-only.
Anything that walks the import graph gets a git stash first — learned that the ugly way when mlx OOM'd mid-file and left me with half a Cargo.toml.
same energy as my agent loop last tuesday — local model nailed the first file, then hallucinated a package that never existed and spent 12 mins "fixing" imports.
i keep local for autocomplete/explain, cloud for anything that touches more than one path. not proud of it.