Agent Skills transfer to goose, but the model decides the payoff: a SkillsBench replication

The finding, with the numbers

A skill pays off through the model twice: once when the model chooses to open the skill, and once when opening it changes what the model does. I measured this by adding goose to SkillsBench (arXiv:2602.12670), the 87-task benchmark that reported curated Agent Skills (modular folders of procedural knowledge) raising agent pass rates from 33.9% to 50.5% on average across OpenHands, Codex, Claude Code, and Gemini CLI. On Claude Opus 4.8, goose's paired uplift is +17.7 percentage points (95% CI [+8.1, +28.1]), statistically indistinguishable from Claude Code's +19.0pp (CI [+6.5, +32.2]); both intervals exclude zero. On GPT-5.5, the identical goose configuration shows no uplift at all (−1.6pp; 7 tasks better, 7 worse), while OpenHands on the same model and the same tasks recovers +8.0pp. The harness and the skills were identical in both goose rows; only the model changed.

I'm an ambassador for the Agentic AI Foundation (AAIF), which stewards goose, and goose wasn't among the harnesses SkillsBench tested. So I ran the benchmark myself: a matched, paired no-skill / with-skill comparison against Claude Code and OpenHands, on Claude Opus 4.8 and GPT-5.5, over 40 of the 87 tasks per pairing. The mechanism the numbers point at is worth stating plainly. The harness controls delivery: whether the skill is discoverable at all. The model controls routing (whether it reaches for the skill) and uptake (whether reaching for it changes the work). A skills uplift number is therefore a property of a model-harness pairing, and mostly of the model.

A note on scope before any more numbers, because it shapes how to read them. This is a practitioner replication rather than a peer-reviewed study. Trials ran single-attempt on one Apple-silicon workstation (the paper used five attempts per task on cloud infrastructure), we report 40 of the benchmark's 87 tasks per pairing (about half the inventory), and the run artifacts aren't published for independent audit. Take the findings as observed patterns and mechanisms, useful for anyone deploying skills, rather than as citable benchmark results. The threats-to-validity section lists everything we know is soft, including a bug of our own that nearly flipped the headline.

One more boundary: this post is about performance. Whether a skill lets an agent self-direct through a procedure without hand-holding is a separate question I'm covering in a companion write-up on SOP-style benchmarks, and I make no autonomy claim from these numbers. SkillsBench measures task reward, nothing more. First the setup, then the Opus 4.8 result, then the GPT-5.5 null, then what it took to get honest numbers.

Why goose, and how we set it up

goose is the interesting omission from the paper. It's ACP-native, extension-based, and governed by a foundation rather than a single vendor. It also differs from the four evaluated harnesses in exactly the mechanics that govern skills: how a skill is discovered, how it reaches the model, and whether the model is nudged to use it. A ranking would tell you little, so we held the tasks, environments, verifiers, and models fixed to see what goose does differently with a skill, and what that teaches anyone deploying skills on any harness.

The benchmark draws from the full SkillsBench inventory: 87 tasks across 8 domains, from software engineering and office work to natural science, industrial systems, finance, mathematics, cybersecurity, and media production. Each task bundles a prompt, a Docker environment, curated skill package(s), and a deterministic verifier that emits a reward in [0,1]. We use the upstream benchflow-ai/skillsbench repo and its bench CLI, which owns environment builds, per-arm skill injection and stripping, verification, trajectory capture, and token and cost accounting. The two arms are the paper's paired protocol: no-skill (skills stripped from the build context) versus with-skill (the task's curated skills deployed into the agent's discovery paths).

Coverage is the honest fine print. For the four pairings we fix n = 40 paired tasks per pairing: the first 40 tasks, sorted by task name, that settled cleanly in both arms. All four report a uniform n over one nearly shared task set (42 distinct tasks appear across the four pairings, and 38 of those appear in all four). That is about half of SkillsBench's 87 tasks. The remainder we didn't carry to completion on every pairing: a handful of environments are arm64-incompatible on Apple silicon, some were still infra-failing at freeze, and the rest fall beyond the 40-task cap. Infra-failed trials are excluded, not scored.

The harness under test is goose v1.41.0, registered into benchflow as a roughly 60-line agent entry: install the CLI in each task container, launch goose acp, map benchflow's provider env onto goose's (GOOSE_PROVIDER/GOOSE_MODEL), and enable the developer, todo, and summon extensions. goose is the only harness here that speaks ACP natively. Claude Code and Codex need adapter shims in benchflow; goose needed registry configuration and nothing else.

For every trial we log mean reward and strict pass (reward = 1.0) per arm; the paired uplift in percentage points with a bootstrap confidence interval (10,000 resamples on the mean-reward delta); a per-task better/worse/tie split (a McNemar-style read on how many tasks the skill actually moved); skill-consultation evidence (string-level traces of SKILL.md access or load_skill calls); tokens, cost, and wall-clock; and a coarse failure taxonomy.

On Opus 4.8 the effect transfers; on GPT-5.5 goose gets nothing

Because we vary both the harness (goose vs the paper's Claude Code and OpenHands) and the model (Opus 4.8 vs GPT-5.5), the cleanest view is one chart. All four pairings are the same n = 40 paired tasks; the two same-model rows in each block are the actual head-to-head.

On Opus 4.8, the effect transfers. Both intervals exclude zero: on a skills-disposed model the effect is real and the same size on both harnesses. goose's +17.7pp is statistically indistinguishable from Claude Code's +19.0pp, and goose has zero regressions: every task the skill moved, it moved up (9 up, 0 down), where Claude Code has two tasks the skill hurt. Strict pass (reward = 1.0) agrees: goose 0.425→0.625, Claude Code 0.350→0.550. For paper context, Claude Code with Opus 4.7 ran 43.0→61.2 (+18.2) and OpenHands with Opus 4.8 ran 45.7→54.1 (+8.4), so our harnesses sit within the paper's Claude-family band on both arms. The trajectories say goose engages skills at least as actively as Claude Code: skill evidence appears in roughly 42% of its with-skill conversations versus roughly 20% for Claude Code (both are string-level floors), and in several trajectories goose imported the skill's helper scripts directly into its own code.

On GPT-5.5, it doesn't transfer for goose. Change the model and the effect that was robust on Opus vanishes: the point estimate goes slightly negative, the interval straddles zero, and the sign split is a dead heat (7 better, 7 worse). OpenHands, which force-injects the skill into its prompt rather than waiting for the model to discover it, recovers a positive point estimate (+8.0pp) that directionally corroborates the paper's row (51.5→67.3), though at single-attempt n=40 its interval still includes zero. We don't report skill evidence for OpenHands: because it injects the container skills path into every prompt, its trajectories reference that path 100% of the time in both arms, so the string-level detector is uninformative there. goose on GPT-5.5 shows about 35% skill evidence. It opens the skill nearly as often as it does on Opus, and converts none of it to reward.

The model mediates twice: adoption, then conversion

Identical harness configuration, identical Summon setup, identical skills on disk; the only change across these two rows is the model.

The obvious read, "adoption drives uplift," turns out to be half the story. GPT-5.5 reaches for the skill at a rate (~35%) comparable to Opus (~42%), yet gets nothing from it. So the model mediates the skills payoff twice: whether it opens the skill (adoption), and whether opening it changes what it does (conversion). Only Opus cleared both bars in this study. An earlier run of ours with a weaker GPT-family model had shown low adoption and a near-zero uplift, which fit the simple adoption story; the GPT-5.5 result is what broke it. This reframes how I read cross-harness uplift tables, including the paper's: they are substantially model-disposition tables, and disposition doesn't reduce to a single adoption rate.

In practice this changes what you measure before trusting a skills A/B. Adoption you can count from trajectories. Conversion you can only see paired: did the tasks where the model opened the skill actually move? A harness can deliver a skill perfectly, the model can read it dutifully, and the number can still come out flat.

Where goose is strong, and where it lags

From the paired tasks and the trajectory record, goose is strongest on well-specified computational work: binary-format parsing, constraint scheduling, data transformation. It runs a lean tool loop with little wasted motion. Its skill uptake is active: when a skill exists, it opens it more often than Claude Code, and on Opus every skill it touched held or improved the score. Two tasks show the pattern at its best: crystallographic-wyckoff-position-analysis, where the skill took goose from 0.45 to a full pass while Claude Code only reached 0.55, and edit-pdf, which the skill unlocked outright (0→1). The full per-task read is in the appendix.

The gaps are just as concrete. goose lags on long-horizon document and media workflows where grinding beats planning (PDF form-filling, video processing). On GPT-5.5 its conversion problem shows up as outright backfires: seven tasks the no-skill arm solved and the with-skill arm didn't, including econ-detrending-correlation and hvac-control (both 1→0). And it averages fewer tool calls per minute than Claude Code on identical budgets, so tight task clocks disadvantage it.

What it took to get honest numbers

Three operational findings that may be more useful to practitioners than the benchmark numbers. Most of them cost us an invalidated run to discover.

Skill discovery is opt-in and silent when it's off

goose discovers skills natively, scanning ~/.config/goose/skills, ~/.claude/skills, and workspace paths, but only when its Summon extension (v1.25+) is enabled. Our first with-skill arm ran with a config that omitted Summon. goose completed every task, never errored, and never saw a single skill. Nothing in the logs flags this. If you are A/B-testing skills on goose or any harness, verify the agent can actually see the skill before trusting your uplift number: read one trajectory end to end.

A delivered skill still needs the model to open it

With Summon enabled and the skills demonstrably in the right directory (we exec'd into live containers to check), a model still only chooses to engage the skill some of the time: about 35% of with-skill conversations on GPT-5.5 and about 42% on Opus. The paper's harnesses differ in how aggressively skills are surfaced (system-prompt injection, prompt nudges naming the available skills, always-on discovery); OpenHands force-injects, goose waits for the model to reach for it. Cross-harness uplift numbers are therefore partly adoption-rate numbers, and, as the conversion result shows, partly conversion-rate numbers on top of that. A with-skill+nudge arm on one model, planned as future work, would separate delivery, adoption, and execution cleanly.

Audit your zeros

A same-day install "optimization" of ours made library-installation failures silent, which stalled roughly 170 goose sessions at startup: zero tool calls, zero messages, verifier reward zero. A first analysis pass scored those zeros as task failures and flipped the entire head-to-head against goose. Those were infrastructure failures rather than the agent failing the task. Re-running the affected arms to completion (2026-07-13) brought every pairing to at least 40 clean paired tasks, which is why this post fixes a uniform n=40 instead of the lopsided freeze counts (goose-Opus 18, Claude Code 55) of an earlier draft. If you run agent benchmarks: never trust an aggregate until you've read a sample of the zeros.

What we'd check next

1. A third with-skill+nudge arm on one model, to separate delivery, adoption, and conversion cleanly.
2. Repeat runs (the paper uses five attempts; we ran one) and the untested half of the task inventory, budget-permitting.
3. Confirming our mean-reward reading against the paper's exact metric definition (the paper reports "pass"; we report both mean reward and strict pass).
4. Publishing the run artifacts so these numbers can be audited independently.
5. Proposing the Summon warning upstream: if skills exist on disk and no extension can surface them, say so in the log.

Threats to validity, including the bug that nearly flipped the result

The result only means something if you know where it's soft. Here is the full list.

1. One machine, one attempt. The paper ran five attempts per task on cloud infrastructure at high concurrency; we ran single attempts on one M4 Max, with repeats planned budget-permitting. Per-task variance is real.
2. Partial coverage: 40 of 87 tasks. We report a uniform n=40 paired per pairing; across all four pairings only 42 distinct tasks appear, about half the benchmark. Absolute rates and uplifts are computed over this subset and could shift on the untested half.
3. arm64 exclusions. Roughly 5 of 87 task environments hardcode x86-only binaries and never build on Apple silicon (documented per-task with build errors). The paper's rows include them; our arms exclude them.
4. Model-version drift. Our GPT-5.5 and Opus 4.8 API snapshots may differ from the paper's evaluation window.
5. Verifier fractional rewards. We report both mean reward and strict pass; the paper reports "pass." We treat mean reward as the comparable quantity, pending confirmation against the paper's exact metric definition.
6. Self-inflicted infra artifacts, disclosed. A same-day install "optimization" of ours made library-installation failures silent, stalling roughly 170 goose sessions at startup (zero tool calls, zero messages). We excluded these as infrastructure failures rather than scoring them as task failures, and re-ran the affected arms to completion on 2026-07-13, bringing every pairing to at least 40 clean paired tasks. The lesson stands: audit your zeros.
7. Skill-evidence detection is string-level. Trajectory greps can under-count subtle skill influence (Summon injects hints into the system prompt, which the trajectory format doesn't fully record) and over-count incidental path mentions. The OpenHands 100%-in-both-arms artifact is exactly this, which is why we don't report its adoption rate.
8. Author affiliation, unpublished artifacts. My AAIF affiliation is disclosed at the top, and the run artifacts aren't published, so you can't audit these numbers yet. Mitigations within that limit: the upstream harness is untouched except the goose registry entry (described below), negative findings are reported alongside positive ones (the GPT-5.5 null, the adoption/conversion gap, two invalidated arms), and no claim here rests on a number smaller than a clear pattern.

Rebuilding this from public parts

The two load-bearing pieces are public: the SkillsBench task inventory and bench CLI (benchflow-ai/skillsbench), and goose itself (block/goose). What we added is thin and fully described: a benchflow agent-registry entry that installs the goose CLI in each task container, launches goose acp, maps BENCHFLOW_PROVIDER_* env onto GOOSE_PROVIDER/GOOSE_MODEL (with special cases for the LiteLLM usage proxy and an OpenAI-compatible local endpoint), and enables the developer, todo, and summon extensions.

With that entry in place, each arm is a single command:

bench eval run --tasks-dir tasks --agent goose --model <m> \
  --skill-mode no-skill|with-skill --sandbox docker

benchflow persists every trial's conversation (acp_trajectory.jsonl, ATIF), timing, tokens, and verifier output.

Where this lands

The Skills effect is real and it transfers to goose, but it is mediated twice: once by the harness (delivery: Summon must be on, paths must be right), and once by the model (disposition to both reach for the skill and act on it). When both align, as with goose on Opus 4.8, the uplift is statistically indistinguishable from Claude Code's on the same 40 tasks, with zero regressions. When the model link breaks, as with the same goose configuration on GPT-5.5, the uplift vanishes even though goose still opens the skill about 35% of the time. Opening a skill and benefiting from it are different bars, and only Opus cleared both here. And when a methodological link breaks (a missing extension, a stalled session scored as a zero), the same benchmark tells you a completely different and wrong story.

For practitioners: evaluate skill ROI per model, never per harness, and re-validate the skill library on every model swap. The rest of the checklist is short. Verify delivery by reading one trajectory end to end. Measure adoption and conversion before judging an uplift number. Audit your zeros. All of this rests on 40 of the 87 tasks, single attempt, on one machine, so: a grain of salt. On the prior question, which mechanism actually delivers a procedure to an agent working unsupervised (a skill versus a tool versus a prompt), the companion argument is in Why Agents Need Skills. Feedback welcome, especially from the SkillsBench authors if we've misread any aspect of the protocol.

Appendix: every task, through a goose lens

One row per task across the 42 distinct tasks in the study. The lens is understanding goose: where it earns its skill uplift, where it already succeeds unaided, and where it misses ground the others take. It isn't a ranking of Claude Code or OpenHands. The clean same-model harness contrast is goose vs Claude Code on Opus 4.8; OpenHands was observed on GPT-5.5, so it enters only as context. Scores are no-skill→with-skill mean verifier reward in [0,1], where 1 is a full pass.