Sorry English.
I pipe llama-3.3-70b through Groq for agent loops because latency look nice on demos. Yesterday: ~900 tok/s for first 4 calls, then hard 429 for ~40 minutes while I still had half a TypeScript refactor open.
Local vLLM on a 4090 is slower but at least it does not ghost me. Anyone using Groq in real CI, or only for demos?
running mistral small:24b on my 3090 via ollama. asked for { "status": "ok" | "error", "latency ms": number } with their JSON mode on. got perfectly valid JSON. every time. fields were state , took m…
pulled deepseek r1:32b on my M2 Pro last night "just to see". Activity Monitor hit 18GB before the first decent answer showed up, fans sounded like a leaf blower aimed at my face. Q4 was usable for a…
Pulled qwen2.5 coder:32b q4 K M into Ollama on an M2 Max (64GB). Activity Monitor said 41GB before the refactor even started. Halfway through rewriting a messy FastAPI router it just… died. No stack…
2 comments
Join the discussion
Log in to comment.
yeah this is the trade. garage 4090 never 429s. i still keep a Groq key for demos because 40 tok/s puts stakeholders to sleep. for real work i stay on
qwen2.5-coder:14bin ollama with num_ctx 16k locked.hit the same wall last week on a LangGraph run. first burst was magic, then every tool call sat on Retry-After. ended up sleeping 2s between nodes which kinda defeats the whole "fast LPU" pitch. demos love Groq. overnight agent jobs... not so much.