vibehacker
News
arXiv (Microsoft Research) ·

Microsoft's Cabra benchmark finds code understanding, not edit volume, is what trips up coding agents

Across 6,840 synthetic call-graph refactoring tasks, six agents in Copilot's harness stayed near-perfect by scripting around size with tools like grep, while plain LLMs degraded; on SWE-bench Verified traces, the count of read and analyze tool calls predicted failures better than lines edited. A new task that merges two near-identical classes finally broke the agents, and code plus data are open at github.com/microsoft/CABRA.

Read at arxiv.org
GitHub Copilot
AI coding assistance across editors, GitHub, and the CLI

More news

View all

Atticus is a free hold-to-talk Mac assistant that runs on Claude Code with no API key

Hold ⌃⌥Space and speak: Apple's on device speech stays local, simple opens and window moves skip AI ( 0.08s), and harder asks go through claude p on your existing Claude Pro/Max plan ( 0.14¢ per ask) while learned shortcuts replay without AI. MIT, needs macOS 26+ and Claude Code; Safe mode sandboxes writes to a scratch folder and prompts before deletes, sends, or buys…

GitHub / SaarthurR (via Show HN)

Expand Fetch turns any URL into agent-ready Markdown with its own vision model, at $1 per 1,000 fetches

Expand's Iris model reads the rendered page, keeps tables and the JSON hidden in the page that other Markdown APIs drop, and links every line back to a saved snapshot; on Expand's own 1,094 question benchmark, agents answered 74% of questions from its output versus 55% for the next best API. It also offers question targeted passages that cut tokens by over 95%, site crawls, and change tracking, and a coding agent can set it up from expand.ai/start.md…

Launch YC (Expand)

Evo² trains one model to both rewrite its own agent harness and learn from the edits, hitting 48.3% on SWE-bench Verified

A single policy proposes harness edits (prompts, tools, context, feedback handling) and reflects on their effect, and those runs then train its weights via SFT plus interleaved GRPO, reaching 82.1% on Polyglot and 48.3% on SWE bench Verified versus 62.6% and 41.0% for the best prior self evolution method. The harness loop alone lifted a GPT 5 mini agent from 24.4% to 48.3%, and the evolved harnesses transferred to other benchmarks without further tuning…

arXiv (Chejian Xu et al.)

Spotted something we missed? Start a thread.