Benchmarks / Published October 2, 2026
Curated skills: 30.5% to 54.3% task success
Task success
Observed pooled rates. Lift is the mean paired task difference.
Without skills: 95% cluster-bootstrap CI 16.7% to 45.8%
With curated skills: 95% cluster-bootstrap CI 37.9% to 69.9%
32 tasks · 189 scored runs · Curated skills, nudged; not comparable to SkillsBench 1.1
Where skills help
Observed domain rate differences. Four sampled tasks per domain. Descriptive, not confidence intervals.
Sonnet 5.5 on a frozen, stratified 32-task SkillsBench v1.1 sample, with three planned trials per arm.
Edge helps your agent find and load specialist skills for the task at hand. We build Edge and ran these studies ourselves. These experiments test the effect of supplying a skill; they do not measure Edge search or automatic retrieval.
Result
29/95 no-skill trials passed; 51/94 curated-skill trials passed. Paired task-mean gain: +24.0 percentage points, 95% CI +10.9 to +38.0. 13 tasks improved, 2 worsened, 17 tied; sign test p=0.0074. Run date: October 1, 2026.
Method
Protocol: the frozen sample and study design below, with public protocol records.
- Sample: 32 tasks, 4 per v1.1 domain, stratified by published difficulty, seed 20261001, frozen in the sample record (commit 09d4f4e) before any model run; selection by metadata only, never by historical lift.
- Pre-flight (checks, leak audit, replacement log): oracle passes, no-op fails, curated-skill leak audit; 6 tasks replaced from their stratum queue and logged.
- Arms: A = no skills; C = the task's curated skills mounted where Claude Code discovers them + one fixed instruction line pointing at them. Same container, model and prompt otherwise. Harbor claude-code agent, local Docker containers, subscription access only. k=3 per task per arm.
- Pass = verifier reward 1.0. Infra trials (verifier timeouts, auth/rate-limit/API errors, env build failures) are excluded and topped up in up to two extra jobs (
.r1,.r2); agent timeouts count as failures. - CIs: pass rate = cluster bootstrap (tasks, then trials), paired delta = task bootstrap of per-task mean(C)-mean(A); 10k resamples. Sign test on tasks with a non-zero delta.
The SkillsBench 1.1 method does not name the skills in the instruction, while this run adds one line pointing at them; a no-instruction arm and a study of the full 87-task roster are in progress, with full-roster trials following sample validation.
Sonnet 5.5 results
| A (no skill) | C (curated skill) | |
|---|---|---|
| Pass rate | 30.5% (29/95) | 54.3% (51/94) |
| 95% CI | [16.7, 45.8] | [37.9, 69.9] |
| Mean raw reward (partial credit) | 0.363 | 0.608 |
| Tasks solved at least once | 13 | 22 |
| Skill-read rate | 0.0% | 83.0% |
| Tokens/trial mean (median) | 140,485 (102,789) | 152,783 (113,786) |
| Output tokens/trial mean | 3,854 | 3,397 |
| Agent wall s mean (median) | 196 (33) | 189 (42) |
| Agent timeouts | 4 | 3 |
Paired over 32 tasks: mean delta C-A = +24.0 pp, 95% CI [10.9, 38.0] pp; C better on 13, worse on 2, tied on 17 (sign test p = 0.0074). Excluding the 2 borderline-leak tasks: A 29.2%, C 51.1%, delta +22.2 pp [9.4, 36.1]. C trials that read a curated skill: 78 (pass 55.1%); did not read: 16 (pass 50.0%). Not causal: reading is self-selected. Published v1.1 aggregate: 33.9% -> 50.5% (+16.6 pp). Here: 30.5% -> 54.3% (+24.0 pp on paired task means).
Per domain
| Domain | Tasks | A | C | Delta pp |
|---|---|---|---|---|
| cybersecurity | 4 | 25.0% | 45.5% | +20.5 |
| finance-economics | 4 | 0.0% | 36.4% | +36.4 |
| industrial-physical-systems | 4 | 16.7% | 50.0% | +33.3 |
| mathematics-or-formal-reasoning | 4 | 33.3% | 33.3% | +0.0 |
| media-content-production | 4 | 33.3% | 91.7% | +58.3 |
| natural-science | 4 | 25.0% | 50.0% | +25.0 |
| office-white-collar | 4 | 83.3% | 83.3% | +0.0 |
| software-engineering | 4 | 25.0% | 41.7% | +16.7 |
Per difficulty
| Difficulty | Tasks | A | C |
|---|---|---|---|
| easy | 2 | 50.0% | 50.0% |
| medium | 19 | 26.3% | 50.0% |
| hard | 11 | 34.4% | 62.5% |
Per task (rewards per trial; C read = trials that opened a curated skill)
| Task | Domain | Diff | A | C | Delta | C read | Why (from trajectories) |
|---|---|---|---|---|---|---|---|
| mario-coin-counting | media-content-production | medium | 000 | 111 | +1.00 | 3/3 | A miscounts coins per frame (counts column wrong); C follows the keyframe-extraction + object-counter template matching and gets exact counts. |
| radar-vital-signs | natural-science | medium | 000 | 111 | +1.00 | 3/3 | A misses heart rate on one recording (harmonic/band choice); C uses the skill's phase-unwrap + band-pass pipeline and passes all recordings. |
| sec-financial-report | finance-economics | hard | 000 | 111 | +1.00 | 3/3 | A answers fail the 0.1% numeric tolerance (wrong 13F aggregation/name matching); C uses 13f-analyzer + fuzzy-name-search and gets all four answers. |
| simpo-code-reproduction | software-engineering | hard | 000 | 111 | +1.00 | 3/3 | Verifier-coupled: the verifier only works if the agent built /opt/py310 (fallback env is unresolvable); C learns that venv convention from the skill, A does not. Not evidence of better reproduction. |
| threejs-structure-parser | media-content-production | medium | 000 | 111 | +1.00 | 3/3 | A part/link meshes do not match ground truth (world transforms not baked); C follows threejs + obj-exporter conventions and matches. |
| manufacturing-fjsp-optimization | industrial-physical-systems | medium | 000 | 110 | +0.67 | 3/3 | A violates the right-shift-only baseline repair policy; the skill spells out that policy, C passes 2/3 (third breaks downtime windows). |
| fix-druid-loophole-cve | cybersecurity | hard | 000 | 01 | +0.50 | 2/2 | All A trials leave a CVE-2021-25646 exploit path open; one C trial (jackson-security skill) blocks it, but hit the agent timeout first; 1 of 2 valid C passes. |
| weighted-gdp-calc | finance-economics | medium | 000 | 10 | +0.50 | 1/2 | Failing trials put wrong-scale values (ratios instead of absolute exports/imports) in the sheet; the one C pass used the xlsx skill, the C failure never opened it. |
| energy-market-pricing | industrial-physical-systems | hard | 101 | 111 | +0.33 | 3/3 | A 2/3 (one run not cost-optimal); C 3/3 with locational-marginal-prices skill. Small effect. |
| manufacturing-codebook-normalization | industrial-physical-systems | medium | 000 | 010 | +0.33 | 3/3 | Both arms fail the same semantic-alignment check; C passes once. Skill helps only marginally. |
| paratransit-routing | mathematics-or-formal-reasoning | hard | 000 | 001 | +0.33 | 3/3 | Both arms mostly hit the 900 s agent timeout or miss reference quality; one C run (OR-Tools pickup-delivery skill) reaches it. |
| software-dependency-audit | cybersecurity | medium | 000 | 010 | +0.33 | 3/3 | Both arms mostly mismatch the ground-truth CSV; one C run (trivy + cvss skills) matches exactly. |
| threejs-to-obj | media-content-production | medium | 001 | 011 | +0.33 | 3/3 | Geometry mismatch in 2/3 A vs 1/3 C; obj-exporter skill helps modestly. |
| dialogue-parser | software-engineering | easy | 000 | 000 | +0.00 | 3/3 | |
| drone-planning-control | industrial-physical-systems | medium | 000 | 000 | +0.00 | 3/3 | |
| exam-block-sequencing | mathematics-or-formal-reasoning | hard | 000 | 000 | +0.00 | 3/3 | |
| exoplanet-detection-period | natural-science | medium | 000 | 000 | +0.00 | 0/3 | |
| fix-erlang-ssh-cve | cybersecurity | hard | 111 | 111 | +0.00 | 0/3 | |
| flink-query | software-engineering | hard | 011 | 011 | +0.00 | 0/3 | |
| invoice-fraud-detection | finance-economics | hard | 00 | 000 | +0.00 | 0/3 | |
| jpg-ocr-stat | office-white-collar | hard | 100 | 100 | +0.00 | 3/3 | |
| mars-clouds-clustering | natural-science | hard | 111 | 111 | +0.00 | 3/3 | |
| multilingual-video-dubbing | media-content-production | medium | 111 | 111 | +0.00 | 3/3 | |
| pddl-tpp-planning | mathematics-or-formal-reasoning | medium | 111 | 111 | +0.00 | 3/3 | |
| pdf-excel-diff | office-white-collar | medium | 111 | 111 | +0.00 | 0/3 | |
| powerlifting-coef-calc | office-white-collar | easy | 111 | 111 | +0.00 | 3/3 | |
| pptx-reference-formatting | office-white-collar | medium | 111 | 111 | +0.00 | 3/3 | |
| quantum-numerical-simulation | natural-science | medium | 000 | 000 | +0.00 | 3/3 | |
| setup-fuzzing-py | cybersecurity | medium | 000 | 000 | +0.00 | 3/3 | |
| xlsx-recover-data | finance-economics | medium | 000 | 000 | +0.00 | 3/3 | |
| pddl-airport-planning | mathematics-or-formal-reasoning | medium | 001 | 000 | -0.33 | 3/3 | C plans use action names the verifier's PDDL reader cannot parse (pddl-skills output format); A passes once. |
| python-scala-translation | software-engineering | medium | 001 | 000 | -0.33 | 3/3 | The hidden ScalaTest spec fails to compile against the agent's API in 5/6 runs; the one A pass guessed the expected signatures. Skill does not address the API contract. |
Failure taxonomy (failed valid trials)
| Category | A | C |
|---|---|---|
| agent timeout | 4 | 1 |
| hidden tests do not compile against agent API | 2 | 3 |
| near miss (>=75% tests) | 26 | 19 |
| no/invalid output | 3 | 2 |
| partial (25-75% tests) | 27 | 18 |
| verifier fallback env unresolvable (no /opt/py310) | 3 | 0 |
| verifier score below 1 | 1 | 0 |
Discussion
- Where skills help: procedural know-how the model does not reliably reconstruct, such as a signal-processing pipeline (radar), template-matching recipes (mario), 13F aggregation with fuzzy issuer matching (sec-financial), three.js world-transform/OBJ export conventions, and a repair policy spelled out only in the skill (fjsp). These tasks flip 0/3 -> 3/3 or 2/3.
- Where skills do not help: tasks with a hidden output contract the skill does not mention. Examples: the hidden ScalaTest API in python-scala-translation, the exact objective reporting in exam-block-sequencing, the growth-value convention in xlsx-recover-data, and the partial-credit drone task that neither arm completes. pddl-airport-planning is a small loss: the skill's plan format produced action names the verifier's parser rejects.
- Caveat, simpo-code-reproduction: the verifier only runs if the agent created
/opt/py310. Its fallback environment (torch==2.1.2+cpuvia uvx) is unresolvable. The skill teaches the/opt/pyXXXvenv convention, so C passes and A cannot. This is a verifier artefact, not evidence of better reproduction. The headline is reported both with and without it. - Read vs unread: C trials that read a skill pass 55%, unread 50% (16 trials). Reading is self-selected, so this is not causal.
- Cost: mean tokens/trial A 140k vs C 153k (+9%). Mean agent wall time is about equal (196 s vs 189 s). Agent timeouts: A 4, C 3.
- Failure shape: most failures in both arms are near misses (>= 75% of checks pass) or partial solutions. The failure mix is similar across arms (near miss 39% of A failures vs 44% of C); C simply has fewer failures. No/invalid output is rare in both arms.
For a trial-level reference, Wilson 95% intervals are 22.2% to 40.4% for A and 44.2% to 64.0% for C. These ignore task clustering; the reported cluster intervals and paired task bootstrap remain primary.
Published SkillsBench comparison
The published v1.1 paper aggregate is 33.9% to 50.5% (+16.6 points), across its model and harness configurations. Ours uses 32 of 87 tasks, a newer model, k=3 and a fixed prompt line pointing to the skills. These are different samples and configurations, so the figures are context, not a model ranking.
Scope and run status
Removing the verifier-coupled simpo-code-reproduction task leaves a paired gain of +21.5 points (95% CI +9.1 to +35.0). Removing the two borderline-leak tasks leaves +22.2 points (CI +9.4 to +36.1). These are sensitivity checks, not replacements for the frozen primary result.
Curated skills were opened in 78/94 valid skill-arm trials (83%). Eight tasks were never fully solved in either arm; seven passed every valid trial in both arms. The skill exposure rate is below our current 90% pilot gate. This is a completed Sonnet result from the earlier frozen protocol, not a claim that current scale gates passed. Three planned trial slots remain unfilled after infrastructure exclusions: one A and two C. Infrastructure-excluded records remain in the download.
Opus 5.5 is queued and quota-paused in the source report. A separate GPT-6.1 Sol study is in progress. Their results will be added after verification.
