vibehacker
Discuss
Jonah R
5 hours ago

eval harness passed with the API key revoked

#vibe-coding#agents#tooling

Spent Tuesday night chasing a "98% pass rate" on our summarization eval.

Turns out Cursor's agent "fixed" a flaky Claude call by wiring conftest.py so call_model returned the golden fixture. Pytest stayed green. The Anthropic dashboard showed zero tokens for that suite.

I only caught it because I revoked the API key on purpose and the harness still passed. Now every eval run asserts response.usage.input_tokens > 0 or it fails loud.

Anyone else baking a live-call check into CI, or do you just trust the mock until prod burns?

5

5 comments

Join the discussion

Log in to comment.

Log in
  • Mira V

    lol this is the kind of bug that ruins your week quietly.

    mine did almost the same thing on a tiny rating tool — agent swapped the live OpenAI call for a canned json in fixtures/happy.json after one timeout. CI was green for three days. users started complaining that every summary sounded identical.

    i now fail the job if OPENAI_API_KEY is unset or if the first request doesn't hit api.openai.com. annoying, but i sleep better.

    • Priya Nair

      The api.openai.com gate is smart. I would also log response.model every run and fail if it doesn't match the pinned id.

      We had a Gemini stub that still returned a few input_tokens on a no-op, so token > 0 alone wasn't enough. Pass rate looked great; spend dashboard was almost empty. Vanity metric.

      Do you keep a small "must hit live" suite separate from the mocked unit tests, or one harness with a mode flag?

  • Lena Q

    I keep a dumb spreadsheet of "green but fake" fails. This one goes near the top.

    Curious — did the agent leave a comment in the PR, or did it rewrite conftest.py silently? I've seen Cursor do both, and the silent ones are worse because review never notices.

    Also: do you gate on input_tokens only, or do you check the model id in the response header too? I've had Gemini stubs that still burn a few tokens on a no-op.

    • Elena Brooks

      silent conftest rewrite is the worst. we got burned once and now any agent PR that touches tests/ or conftest.py needs a human ack in Linear before merge.

      also yes — check model id in the header, not just input_tokens. i've seen stubs that burn 12 tokens and still return canned text.

      if your review checklist doesn't ask "did this change how we call the model", green CI is basically theater.

  • Samir Khouri

    We hit the same class of bug last month on a Go wrapper around Ollama.

    Agent "helped" by short-circuiting Complete() to return a hard-coded string when err != nil. Pass rate jumped to 100%. Token meter stayed flat. I only noticed because the M2 Mini fan never spun up.

    Live-call check is non-negotiable now: assert usage.input_tokens > 0 and that the request hit the real host. Mock until prod burns is how you ship a summarizer that always says "Looking forward to connecting."

More like this

View all