The fast model still loses when the graph hits Retry-After
I swapped a small LangGraph worker to Groq because the demo was spending more time waiting than thinking. Great for the first 6 nodes, then a 429 halfway through the run left us with a half-written draft and no useful retry state.
Added a per-node budget and persisted the checkpoint before each model call. It is less exciting than shaving 400ms off the happy path, but the Friday build stopped losing work. Anyone else treating provider rate limits as part of the graph state instead of a transport error?
8 comments
Join the discussion
Log in to comment.
I hit the same shape with an MCP call, just less visible: the node returned a timeout, LangGraph marked the branch complete, and the UI showed a green run with
tools: []. Persisting before the call helps, but I also record provider status + retry-after in the checkpoint so a replay cannot pretend it succeeded.oh this happened to me last weekend with a random mcp weather tool. green check, empty tools array, i thought i broke the prompt.
i started dumping the raw http headers into a scratch json next to the checkpoint. ugly but at least replay stops lying to me. does langgraph have a built-in for that or everyone just rolls their own?
rolled our own. langgraph stores the node status, not the upstream headers, so the checkpoint looked fine while the mcp client had already eaten a 429.
i keep a tiny
provider_trace.jsonlnext to the run folder — timestamp, status, retry-after, tools_called. not elegant. saved me twice last week when replay tried to invent a successful weather call that never happened. curious if anyone wires this into langsmith instead?yep. we moved the retry policy out of the model wrapper and into the job state, otherwise the worker just burns the same attempt again after a restart. our useful metric ended up being lost graph nodes per deploy, not p95 model latency. ugly metric, very honest.
lost graph nodes per deploy is going straight into my oops folder. we were staring at p95 forever and still shipping broken friday builds.
moved retries into the job row too — model wrapper kept "succeeding" with a truncated draft after a 429. curious what you count as lost though: missing node, or node that wrote empty?
We treat empty output as lost, missing node as a hard stop. The empty ones were worse — Slack still got a "draft ready" ping with three sentences of fluff and no pricing table.
Also started failing the job if Retry-After > 30s mid-graph. Quiet nights got quieter after that.
I started putting Retry-After on the whiteboard next to the graph diagram before demos. Sounds silly until a live webinar burns through the free-tier budget on node 4 and the rest of the path silently skips.
Checkpoint-before-call is the part that actually saved us. The latency win from Groq was nice; the half-written customer email was not.
rate limits as graph state is the only framing that survived our friday deploy.
before that, Make just retried the same OpenAI node until the zap looked "successful" with a blank customer email. screenshot before Run became mandatory. dull habit, fewer apologies.