How we test. Measure the difference.

How we test

We screen domains separately before the final study. The screening tasks show where the no-skills agent has substantial room to improve. Fresh tasks from qualifying domains are frozen before either final arm runs, and every final outcome is published.

The two arms

Both arms use the same task checkout, ordinary tools and resource limits. Web search and subagents are disabled. The subject agent never receives provider credentials; model traffic goes through a host broker.

Tasks

The separate screening set contains 33 tasks. We run the no-skills arm on all of them and select domains where it fails at least two of three valid tasks. Screening tasks never count toward the headline. If fewer than three credible domains qualify, we use the predeclared fallback: externally sourced hard tasks with fixed selection criteria and objective verifiers.

Before either final arm runs, we freeze at least 12 fresh held-out tasks from qualifying domains. A task is invalid if its hidden check demands an interface detail absent from the brief. Invalid originals are documented and excluded; repaired versions may help development but cannot enter the held-out score.

SkillsBench is a separate later study. No SkillsBench task or outcome enters this pool.

Fairness rules

• No skill is written or edited after the tasks are known.
• Both arms receive the same task prompt, base checkout, ordinary tools, network policy and resource limits. Edge access is the intended treatment difference.
• Domain selection uses separate screening data. We do not search for individual tasks the baseline failed and then add them to the final set.
• Final task eligibility and membership are fixed before either final arm runs. Retrieval failures, baseline wins and Edge losses remain visible.
• One primary solve attempt per task and arm. An infrastructure pause can be repaired and resumed without silently replacing tasks or selecting the best retry.

Verification

Frozen, task-specific tests or format checks establish pass or fail. An LLM judge does not decide task success. Missing or invalid verifier results stay unresolved until verification is recovered; they are never silently counted as passes, failures or exclusions.

What we report

We publish every frozen held-out task and both arm outcomes: objective result, verifier evidence, attempts and failure reason. Paired Edge wins, ties and losses include both-pass and both-fail ties. Screening and pilots appear separately from the headline.

For Edge runs we also show whether the agent searched, received a nonempty verified skill payload, and followed applicable guidance. Search or load success never removes a task from the denominator. We report model tokens for the agent and Edge provider where available, with coverage clearly marked if provider usage cannot be verified.

Current status

Domain screening and task selection are underway. No held-out task result is published here yet. The final task list, runtime and verifier records will be linked when frozen.

Limitations

Results will describe the tested tasks, model and catalog snapshot. The evaluation set is small within each domain. Every loss and unresolved infrastructure case will be shown alongside any aggregate.

Reproduce