AT
Ava Thompson
12 hours ago
claude wrote a rubric that graded citations that don't exist
I asked Claude (Sonnet, $20/mo Pro) to draft a grading rubric for an 8th grade research essay. Looked sharp. Five criteria, point weights, even sample comments.
Then I spot-checked one "exemplar" citation against the school library catalog. Author didn't exist. Journal issue was from 2027. The rubric still awarded full marks for "accurate sourcing."
I almost emailed it to three teachers on Friday. Now every AI rubric goes through a fake-citation hunt first. Anyone else catching invented homework scaffolding?
4 comments
Join the discussion
Log in to comment.
same energy as meeting notes that invent action items. if the brief looks polished and the sources are wrong, trust the sources. i keep a graveyard of cancelled $20/mo tools for this exact reason.
measure it. on a 20-essay sample i got 6 invented DOIs and 2 authors with plausible names who publish nowhere. coverage without a citation check is worse than a sparse rubric. false-positive rate beats vibes every time.
same numbers energy on the growth side. I ran a 15-slide client deck through Claude for "source polish" and it invented two McKinsey quotes with real-looking years.
cancelled the $20/mo seat that night. fake sources in a paid deck cost more than the subscription. citation check or it doesn't leave the draft folder.
I almost shipped the same thing last month for a Lisbon tutoring pilot. Sonnet drafted a lovely "source quality" criterion, then pointed kids at a journal that doesn't exist until 2028.
Now every AI rubric goes through a DOI ping first. Looks polished != is safe. Did you catch it before the teachers emailed, or after?