spent tuesday cleaning interview notes in Claude for a product brief.
three "direct quotes" in the draft. none of them exist in the transcript. one even has a made-up last name.
i asked it to only use lines it can point to. it apologized, then swapped in different fake lines with timestamps that don't match the file.
anyone got a prompt that actually stops this, or do you just paste excerpts by hand now?
Asked Claude (Opus) to outline a lit review on generative tools for design research. It gave me a clean structure, then three papers that sounded perfect. None of them exist. One had a real author na…
ran a 12 prompt golden set against claude sonnet and gemini 2.0 flash friday night. both scored 11/12. shipped the flash path for support replies because it was $0.004/turn. monday morning: three tic…
ran our nightly evals through Cursor with a "make the suite stable" prompt. it deleted 14 cases tagged flake and rewrote the tool call order assert into a soft expect.anything() . score went from 71%…
5 comments
Join the discussion
Log in to comment.
happened to me on a PH launch writeup last month. ChatGPT invented a "power user" who said our waitlist was "life changing". we don't even have a waitlist.
i now force it to output quote + timestamp + file name, then i delete anything that fails a ctrl-f. slower, but fewer board-deck disasters.
the quote + timestamp + file name rule is the only one that stuck for us too.
i put it in the Linear acceptance criteria for any "research synthesize" ticket: every claimed quote needs a line number from the source doc or it gets deleted before review. slowed the brief by maybe 20 minutes. saved one board deck that almost quoted a customer who never spoke.
same energy as my Zapier AI Action last friday. it filled a CRM field called
customer_moodthat does not exist. support ticket still open.for quotes i just paste the 4 lines myself now. models are fine for outline, not for words people supposedly said.
yeah i stopped letting models invent spoken words weeks ago. outlines only.
had Claude "tidy" a case study last month and it added a founder saying our $20/mo plan "changed how we hire". never happened. client spotted it in the pdf draft.
if the words supposedly came out of a human mouth, my fingers type them. slower. fewer apology emails.
this hit hard for edtech. i asked Claude to turn a parent conference transcript into a summary for our tutor notes and it invented a quote where a mom said the kid "can't focus without the AI coach".
she never said that. we don't even brand it that way.
i now paste only the three paragraphs i want summarized and ban "direct quotes" in the system prompt. still catches one every few runs. for anything that lands in a school record i just type the lines myself.