PC
Pine Cascade
1 day ago
search agent ranked our stale SOAP blog above the API docs
spent tuesday wiring a tiny retrieval loop over our help center. the agent kept quoting a 2023 blog post that says we still support SOAP. we killed SOAP in january.
i added a last_updated filter and it still pulled that page because the markdown frontmatter said 2024-11 (someone never bumped it). trust nothing without a crawl date, apparently.
anyone pinning a hard max age on retrieval or are you all just yelling at the model?
7 comments
Join the discussion
Log in to comment.
this is why i stopped indexing blog posts into the same corpus as docs. we split them into two indexes and the agent has to ask for
docsormarketingexplicitly.still catches the frontmatter lies though — i run a nightly job that overwrites
last_updatedfrom git blame. ugly, but the SOAP incident never happened twice.split indexes is smart. did you also block the agent from querying marketing unless the ticket literally says "pricing" or "blog"?
we tried that and the model still sneaked in via a "related reading" chunk that lived inside the docs index. had to strip outbound links from docs before embed.
we got burned the other way: agent ignored a page updated yesterday because the crawl cache was 9 days old. max age alone is not enough if your indexer is asleep.
now i fail the draft if any citation is older than 90 days OR if crawl_ts is older than 48h. loud, but sales stopped sending customers to dead features.
the frontmatter lie is half the story. we also caught docs pages where the H1 still said "SOAP gateway" after the body was rewritten for REST — embeddings latched onto the title.
i strip titles to a slug + date before embed now. looks terrible in the citation UI. sales stopped linking dead protocols though.
we pin a hard max age of 60 days AND require crawl_ts from the indexer, not frontmatter. last month the agent still found a "how to call our SOAP gateway" page because someone had republished it as a redirect with a fresh date.
now redirects inherit the target's crawl_ts. messy, but the SOAP ghost finally died.
split indexes helped us too but a blog post still snuck in via a sitemap the crawler treated as docs.
now marketing URLs get a corpus=marketing tag at crawl time and the agent query tool refuses that tag unless the ticket category is literally content. ugly gate. works.
we put the dollar ceiling in the tool schema after a similar mess — max credit amount as a required arg with a hard 25 default the model cannot override.
prompt comments are fiction. if Stripe will accept the call, the agent will make it. also capped how many apology sentences it can emit before it has to escalate; fourteen is a UX bug, not empathy.