Benchmarks / Published October 2, 2026
Personal skills: 17% to 58% task success
Task success
Observed pooled rates. Lift is the mean paired task difference.
Without skills: 95% Wilson trial CI 7% to 36%
With workflow skills: 95% Wilson trial CI 39% to 76%
12 tasks · 48 scored runs · Exploratory pilot, study-written tasks, mounted skills
Sonnet 5.5 on 12 study-written personal workflow tasks, with two trials per arm.
Edge helps your agent find and load specialist skills for the task at hand. We build Edge and ran these studies ourselves. These experiments test the effect of supplying a skill; they do not measure Edge search or automatic retrieval.
Result
With personal skills: 14/24 (58%) fully passed, Wilson 95% CI 39% to 76%. Without: 4/24 (17%), CI 7% to 36%. Paired task-clustered gain: +42 percentage points, bootstrap 95% CI +8 to +71. Partial credit: 165/242 (68%) to 217/242 (90%). Run date: October 1, 2026.
Method and sample
Twelve tasks written for this study, covering five frequently used personal skills: Buffer scheduling, LinkedIn writing, static site deployment, Edge copy and Gmail operations. Sonnet 5.5 via Harbor Claude Code 2.1.286, k=2 per task per arm, 48 trials total. All services were offline mocks; no real credentials or messages were used. The no-skill arm had no memory, CLAUDE.md or tool hints. The skill arm mounted only the skill under test, with the same task prompt and no nudge. All 24 skill-arm trials loaded SKILL.md through the Skill tool.
121 deterministic checks per set of 12 tasks; each trial passes only if all its task checks pass. User standing rules are included because personal skills carry those rules. Checks were written after reading the skills: these are study-written tasks, not held-out real requests. Oracle passed 12/12; no-op failed 12/12. Freeze: commit d743b84, tag s2-frozen, before either arm ran.
Check categories
| Category | Checks per 12-task set | A failed (k=2) | Skill failed (k=2) |
|---|---|---|---|
| Task completion | 37 | 31/74 | 11/74 |
| User standing constraints | 35 | 28/70 | 5/70 |
| Workflow procedures | 15 | 10/30 | 8/30 |
| Private context | 12 | 8/24 | 1/24 |
| Prohibited side effects | 22 | 0/44 | 0/44 |
Task-level outcomes
| task | Request | A | S |
|---|---|---|---|
| buffer-ig-reel | Schedule an Instagram reel with hashtags in its first comment | 0 0 | 0 1 |
| buffer-li-reschedule | Move a LinkedIn post while preserving media and metadata | 0 0 | 0 1 |
| buffer-x-video-reply | Schedule an X video post with the skill link in a reply | 0 0 | 0 1 |
| edge-whatsapp-cursor | Draft a founders-group announcement with a setup path | 0 0 | 1 1 |
| edge-x-launch-shazam | Draft an X launch post in the requested rotation format | 0 0 | 1 1 |
| getedge-add-skill-page | Build a catalog page for a merged skill on a branch | 1 1 | 1 1 |
| getedge-rename-title | Rename a page title and refresh its preview card | 0 0 | 0 0 |
| gmail-draft-pilot | Draft a threaded email reply in the correct mailbox | 0 0 | 1 0 |
| gmail-reply-partnership | Reply to everyone in an email thread from its mailbox | 0 0 | 0 0 |
| li-hackathon-recap | Turn voice-note material into a LinkedIn recap | 1 1 | 0 0 |
| li-launch-from-notes | Write a LinkedIn launch post from raw notes | 0 0 | 1 1 |
| li-less-techy | Rewrite a LinkedIn post in plain language | 0 0 | 1 1 |
Failure patterns and sensitivity
Without a skill, the agent never reached the Buffer or Gmail tools. Writing tasks missed standing rules about links, capitalization, length and layout. Site build gates helped both arms equally. The skill arm still dropped an image during a reschedule; both arms missed a social-card refresh and a reply-all recipient.
Two skill trials correctly refused a URL pre-check because the mock environment lacked the required host; primary scores retain these failures. A capitalization check penalized a meme example present in the skill itself. Dropping only that check raises skill success to 16/24; excluding only the two blocked pre-check trials gives 14/22. Neither replaces the frozen primary result.
Three tasks were never fully solved in either arm; one task always passed in both arms. No trial raised a harness exception. The small sample gives wide intervals. Skills here encode tool discovery, private context and personal rules, so the result does not establish better general engineering or Edge retrieval. A larger held-out S2c replication is in progress; no S2c outcome is included here.
Evidence
Protocol, report and results on GitHub · Redacted per-trial results and check outcomes · Public summary
Private names, emails, account identifiers, diagnostic values and transcripts are omitted. Task identifiers, scalar metrics, check outcomes and check categories are retained. No private task fixtures or skill contents are published.
