KN
Kai Nakamura
13 hours ago
Groq was so fast I thought my eval harness was broken
Groq
Ultra-fast LLM inference on custom LPU chips
Switched our nightly prompt eval from OpenAI to llama-3.3-70b on Groq last week.
First run finished in 41 seconds. Same suite used to take ~11 minutes. I spent twenty minutes poking the cache layer before I accepted the API was just… that quick.
Bill for the whole suite: $1.12. Still not sure I trust quality on the weird edge cases though — anyone comparing Groq vs Fireworks on the same prompts?
5 comments
Join the discussion
Log in to comment.
ok but whats the rate limit like when you burst? we hit 429s mid-demo on a free key and looked like clowns. $1.12 is cute until you parallelize.
hit the same 429 wall on a free key during a live demo last tuesday. switched to a paid key mid-call which felt extremely professional.
if youre bursting evals, request a higher limit early. the dashboard underplays how fast you burn tokens when you go parallel.
Same shock here. We moved a staging bot onto Groq for a latency demo and suddenly product stopped asking why the spinner sat for 4s.
Quality-wise I still route hard reasoning to Claude though. Groq for the interactive path.
curious how you decide the cutover — is it latency budget, or do you have a cheap classifier that routes "hard" prompts to Claude?
i keep seeing teams split interactive vs reasoning like that and then never write down the rule. would love a one-liner if you have one.
I ran the same 200-prompt suite on Groq llama-3.3-70b and Fireworks llama-3.3-70b last Friday.
Groq: 48s, $1.09. Fireworks: ~3m20s, $0.84. Quality on our weird JSON-schema cases was basically a coin flip — Fireworks won 7 of the 12 failures we care about.
So yeah, speed is real. I wouldnt fire OpenAI for the hard path yet.