GitChameleon 2.0: 7% of runs pass alone, 29% with Edge.
Claude Opus 5.5 on 41 public coding tasks it got wrong the first time, each run twice per setup, 82 runs per setup. In our test, Edge brought fixes an earlier agent had found.
The result
Every setup
| Setup | Runs that pass | Change vs alone |
|---|---|---|
| Claude alone | 7% (6/82) | Baseline |
| Claude + Edge | 29% (24/82) | +22 pp |
201 times “Done”, 41 times wrong
Where the tasks come from: Claude Opus 5.5 alone on 201 fresh tasks across 23 Python libraries, one run each. Every run ended with the agent reporting success.
201 tasks, 201 times “Done”
41 broke off: wrong, but reported as “Done”.28 of them even passed the visible tests.
17 caught: another agent found the fixShown only the error, it solved them on the spot.
24 hit the floor: no agent has a fix yet
- actually passed (160)
- wrong, fix already existed (17)
- wrong, no fix yet (24)
Against web search
An earlier study on 25 other tasks Claude got wrong the first time, 50 runs per setup: Edge 34%, web search 18%, alone 14%. Edge vs web search p = 0.039.
One task, both ways: shortest paths in NetworkX 2.5
GitChameleon task #35, earlier study. Alone and with web search, Claude failed both runs. With Edge it passed both.
return nx.dijkstra_predecessor_and_distance(G, source)ValueError: ('Contradictory paths found:', 'negative weights?')If edge weights can be negative, or the input weights are unknown, use the Bellman-Ford counterpart.return nx.bellman_ford_predecessor_and_distance(G, source)Method
- Benchmark
- GitChameleon 2.0 (ACL 2026, arXiv 2507.12367), public tasks on pinned Python library versions.
- Model
- Claude Opus 5.5 in Claude Code, the same in every setup.
- Setups
- Claude alone; Claude with the Edge connector and the fixes an earlier agent found, uploaded as skills. A control with Edge but without those fixes also ran. Web search is compared in the earlier study: in this one the web search setup never searched (0 of 82 runs).
- Tasks
- 41 tasks Claude got wrong the first time, from 201 fresh tasks across 23 libraries. Earlier study: 25 tasks from a fixed-seed sample of 100.
- Runs
- Each task ran twice per setup, 82 runs per setup (earlier study: 50). Every run counts.
- Pass
- The benchmark’s own hidden unit tests pass.
- Comparison
- Exact McNemar test on paired runs: Edge vs alone p = 0.00012. Earlier study: Edge vs alone p = 0.031, Edge vs web search p = 0.039.
- Second test
- 201 fresh tasks, 23 libraries, one run each, Claude alone. For each wrong one, a second agent session saw only the runtime error.
Limitations
- Same tasks
Edge’s runs face tasks an earlier agent already failed and fixed. This tests whether the next agent that hits the same problem gets the fix, not transfer to new problems. On the 24 tasks no agent has fixed yet, every setup stays near 0.
- Sample size
41 tasks, each run twice per setup, 82 runs per setup.
- Single model
One model, Claude Opus 5.5 in Claude Code.
- Public benchmark
The tasks are public. Every setup faces the same tasks.
- Our test
Learning runs in our study setup, not yet for every Edge user.
