Do Agent Skills Transfer to goose? A Practitioner's Replication of SkillsBench

Introduction

SkillsBench (arXiv:2602.12670) made a clean claim: curated Agent Skills — modular folders of procedural knowledge — raise agent pass rates from 33.9% to 50.5% on average across 18 model-harness configurations spanning OpenHands, Codex, Claude Code, and Gemini CLI. The subtext most people took away is that a smaller model with Skills can match a bigger model without them. But every number in that table is mediated by a harness — the agent loop that decides whether a skill is even discovered, how it's surfaced to the model, and whether the model is nudged to use it. The paper tested four harnesses. It did not test goose, and goose behaves differently from all four in exactly the ways that matter for Skills.

So we added it: the full 87-task inventory, the same Docker environments, the same deterministic verifiers, the same paired no-skill / with-skill protocol — with goose driven over its native Agent Client Protocol (ACP) across three model classes plus a secondary datapoint. Then we did the thing benchmark tables usually skip: we read the trajectories. Whether the agent actually opened the skill, how many tokens each arm burned, and why a given trial scored zero.

Here's the short version. Skill delivery is harness-specific plumbing — goose only discovers Skills when its Summon extension is on, and silently ignores them otherwise. Delivery is not adoption — with discovery working, goose + GPT-5.2 chose to engage the provided skill in only ~12% of tasks, and its uplift was correspondingly small (+1.9pp). And Skills act as exploration compression — even at low adoption, the with-skill arm used 34% fewer tokens at equal-or-better accuracy. The headline lands on Opus 4.8: goose's Skills uplift (+22.6pp) is statistically indistinguishable from Claude Code's (+25.3pp) at near-identical absolute pass rates. The effect fully transfers to goose when the model is disposed to use skills.

What's inside:

• Three operational findings that cost us an invalidated run each — and are more useful than any single pass rate
• The head-to-head: goose vs Claude Code on Opus 4.8, at parity
• Why cross-harness uplift tables are substantially adoption-rate tables
• A candid threats-to-validity section, including a self-inflicted bug that flipped the result until we audited our zeros

A note on scope, up front, because it shapes how to read the rest: this is a practitioner study, not a peer-reviewed evaluation. Trials ran single-attempt on one Apple-silicon workstation (the original work used five attempts on cloud infra), a few tasks were excluded for arm64 Docker incompatibilities, and the evaluation artifacts are not published for independent verification. Take the numbers as observed patterns and mechanisms, not citable benchmark results. Section 5 enumerates the threats. Disclosure: the author is an AAIF ambassador.

Why this comparison

Agent Skills are having a moment. SkillsBench formalized the intuition that packaged procedural knowledge beats raw model capability on specialized workflows, and reported that smaller models with Skills can match larger models without them. That result is real. It is also, every cell of it, filtered through a harness that decides whether a skill is discovered, how it reaches the model, and whether the model is pushed to use it.

goose is the interesting omission. It is ACP-native, extension-based, and the only major open-source agent governed by a foundation — and it differs from all four evaluated harnesses in precisely the mechanics that govern Skills. So the question here is deliberately not "is goose better or worse." It is: when you hold the tasks, environments, verifiers, and models fixed, what does goose do differently with a skill — and what does that teach anyone deploying skills on any harness?

Setup

The benchmark is the full SkillsBench inventory: 87 tasks across 8 domains — software engineering, office / white-collar, natural science, industrial-physical systems, finance / economics, mathematics / formal reasoning, cybersecurity, and media / content production. Each task bundles a prompt, a Docker environment, curated skill package(s), and a deterministic verifier that emits a reward in [0,1]. We use the upstream benchflow-ai/skillsbench repo and its bench CLI, which owns environment builds, per-arm skill injection/stripping, verification, trajectory capture, and token/cost accounting.

The two arms are the paper's paired protocol: no-skill (skills stripped from the build context) versus with-skill (the task's curated skills deployed into the agent's discovery paths). The harness under test is goose v1.41.0, registered into benchflow as a ~60-line agent entry: install the CLI in each task container, launch goose acp, map benchflow's provider env onto goose's (GOOSE_PROVIDER/GOOSE_MODEL), and enable the developer, todo, and summon extensions. goose is the only harness here that speaks ACP natively — Claude Code and Codex need adapter shims in benchflow; goose needed none, just registry configuration.

For every trial we log mean reward and strict pass (reward = 1.0) per arm; skill-consultation evidence (string-level traces of SKILL.md access or load_skill calls in the trajectory); steps, tool calls, tokens, cost, wall-clock; and a failure taxonomy (solved / partial / failed-engaged / failed-barely-engaged / infra-error), with per-domain breakdowns.

What we learned before any pass rate: skills plumbing is harness-specific

Three operational findings that are, arguably, more useful than the benchmark numbers — each cost us an invalidated run to discover.

Discovery is opt-in, and silent when it's off

goose discovers skills natively — it scans ~/.config/goose/skills, ~/.claude/skills, and workspace paths — but only via its Summon extension (v1.25+). Our first with-skills arm ran with a config that omitted Summon. goose completed every task, never errored, and never saw a single skill. Nothing in the logs flags this. If you are A/B-testing skills on goose — or any harness — verify the agent can actually see the skill before trusting your uplift number. Read one trajectory end to end.

Delivery is not adoption

With Summon enabled and skills demonstrably in the right directory (we exec'd into live containers to confirm), goose + GPT-5.2 chose to engage the skill in only ~12% of with-skill conversations. The paper's harnesses differ in how aggressively they surface skills — system-prompt injection, prompt nudges that name the available skills, native always-on discovery — and benchflow itself ships a BENCHFLOW_SKILL_NUDGE mechanism for exactly this reason. So cross-harness uplift numbers are, in part, adoption-rate numbers. The natural next arm is a third with-skill+nudge condition on one model, to separate delivery, adoption, and execution.

Local models fail at the template layer, not the size layer

For the Ollama arm, qwen2.5-coder:7b — a capable coding model — reliably emitted tool calls as plain-text JSON: unparsed, zero tool executions, automatic failure, under both the OpenAI-compatible endpoint and goose's native Ollama provider. Its sibling qwen2.5:7b, whose Ollama chat template has proper tool-call support, worked immediately. Local-model agentic viability hinges on the tool-calling template, not the parameter count.

Results

All numbers are mean verifier reward over paired tasks (the same task present in both arms), single attempt, on our machine. "n" is the paired-task count at freeze. Infra-failed trials — including a goose session-stall bug we introduced ourselves (see §5) — are excluded, not scored.

goose × GPT-5.2 (complete; 84 valid trials/arm)

Small uplift (+1.9pp), large token compression (−34%). Skill-consultation evidence shows up in only ~5–12% of conversations: GPT-5.2 rarely chooses to open the skill. That the with-skill arm still costs less and scores no worse is the first hint of what a skill does even when the model half-ignores it — it compresses exploration.

goose vs Claude Code × Claude Opus 4.8 — the head-to-head

Paper context: Claude Code × Opus 4.7 ran 43.0→61.2 (+18.2); OpenHands × Opus 4.8 ran 45.7→54.1 (+8.4). Our two harnesses sit within a few points of each other in both arms, and both show uplifts at or above the paper's Claude-family rows. The headline: with a skills-disposed model, goose's Skills uplift matches the best harness in the paper's table, at parity with Claude Code on absolute rates. Skill-consultation evidence: goose 31–47% of with-skill conversations versus Claude Code 21–27% (string-level floors) — goose engages skills at least as actively.

A per-task LLM-assisted comparison (claude-haiku-4-5 as analyst, activity-valid tasks only) came out goose-stronger 4, similar 7, mixed 4, goose-weaker 4 — parity, echoing the aggregate.

goose vs OpenHands × GPT-5.5 (partial)

OpenHands directionally corroborates the paper's row (51.5→67.3). goose's GPT-5.5 arms were the primary casualty of our session-stall bug (§5); completing them is the first item of future work. From its 26 clean with-skill trials, goose × GPT-5.5 scored 0.298 with 38% skill evidence — consistent with adoption being model-mediated — but we don't report a paired delta on this coverage.

Adoption is model-disposition, not harness luck

Same goose, same Summon config, same skills on disk. The only thing that changes across these two rows is the model.

The model decides whether to reach for the skill. Opus opens it roughly 4–8× more often than GPT-5.2 under identical goose configuration, and the uplift tracks the adoption almost exactly. This single contrast explains most of the variance we observed, and it reframes cross-harness uplift tables: they are substantially adoption-rate tables. A harness can deliver a skill perfectly and still show a flat uplift, not because delivery failed but because the model kept its own counsel.

Where goose shines, and where it doesn't

From the clean Opus pairs, the GPT-5.2 pairs, and the trajectory record, a consistent shape emerges — worth naming because it's the part that generalizes past this one benchmark.

Where it shines. Well-specified computational tasks — binary-format parsing, constraint scheduling, data transformation — where goose runs a lean tool loop with no wasted motion. Active skill uptake: when a skill exists, goose opens it more often than Claude Code and, in several trajectories, imports the skill's helper scripts directly into its own code. And persistence: goose's failures are engaged-but-failed rather than never-started, and its with-skill arms consistently cost less than the no-skill arms — the exploration-compression effect again.

Where it doesn't, yet. Long-horizon document and media workflows where grinding beats planning — PDF form-filling, video processing. GPT-family adoption: with GPT-5.2 and GPT-5.5, goose keeps its own counsel and ignores available skills far more than it does with Claude models. And per-step pace: goose averages fewer tool calls per minute than Claude Code on identical budgets, so tight task clocks disadvantage it.

Threats to validity

The result only means something if you know where it's soft. Here is the full list, including the one that nearly published the wrong conclusion.

1. One machine, one attempt. The paper ran n=5 attempts on cloud infra at high concurrency; we ran n=1 on a single M4 Max (repeats planned, budget-permitting). Variance on individual tasks is real.
2. arm64 exclusions. Roughly 5 of 87 task environments hardcode x86-only binaries and never build on Apple silicon (documented per-task with build errors). The paper's rows include them; our arms exclude them, shifting absolute rates by a few points at most.
3. Model-version drift. Our GPT-5.5 / Opus-4.7 API snapshots may differ from the paper's evaluation window.
4. Verifier fractional rewards. We report both mean reward and strict pass; the paper reports "pass." We treat mean reward as the comparable quantity, pending confirmation against the paper's exact metric definition.
5. Skill-evidence detection is string-level. Trajectory greps can under-count subtle skill influence (Summon injects hints into the system prompt, which the trajectory format doesn't fully record) and over-count incidental path mentions.
6. Self-inflicted infra artifacts, disclosed. A same-day install "optimization" of ours made library-installation failures silent, which stalled ~170 goose sessions at startup (zero tool calls, zero messages). We excluded these as infrastructure failures — not scored as task failures — which is why goose's Opus n is 18 rather than 55+. A first analysis pass that scored stalls as failures flipped the head-to-head. Verifying why an agent scored zero — stall versus genuine failure — changed the story completely. If you run agent benchmarks: audit your zeros.
7. Author affiliation, unpublished artifacts. AAIF ambassador, and the run artifacts aren't published — you cannot audit these numbers. Mitigations within that limit: the upstream harness is untouched except the goose registry entry (described in §6), negative findings are reported (the adoption gap, two invalidated arms) alongside positive ones, and no claim here rests on a number smaller than a clear pattern.

Rebuilding this from public parts

The two load-bearing pieces are public: the SkillsBench task inventory and bench CLI (benchflow-ai/skillsbench), and goose itself (block/goose). What we added is thin and fully described: a benchflow agent-registry entry that installs the goose CLI in each task container, launches goose acp, maps BENCHFLOW_PROVIDER_* env onto GOOSE_PROVIDER/GOOSE_MODEL (with special cases for the LiteLLM usage proxy and an OpenAI-compatible local endpoint), and enables the developer, todo, and summon extensions.

With that entry in place, each arm is a single command:

bench eval run --tasks-dir tasks --agent goose --model <m> \
  --skill-mode no-skill|with-skill --sandbox docker

benchflow persists every trial's conversation (acp_trajectory.jsonl, ATIF), timing, tokens, and verifier output.

Conclusion

The Skills effect is real and it transfers to goose — but it is mediated twice. Once by the harness (delivery: Summon must be on, paths must be right), and once by the model (adoption: Opus reaches for skills ~4–8× more often than GPT-5.2 under identical goose configuration). When both align, goose's uplift — +22.6pp on Opus 4.8 — matches the strongest harness row in the SkillsBench table, at parity with Claude Code on absolute rates. When either link breaks — a missing extension, an indifferent model, or a stalled session scored as a zero — the same benchmark tells you a completely different, and wrong, story.

So the "do this Monday" version, for anyone deploying skills on any harness: verify delivery in one trajectory, measure adoption before you judge uplift, and audit your zeros. On which mechanism actually delivers a procedure to an agent that has no one watching — a skill versus a tool versus a prompt — that's the companion argument in Why Agents Need Skills.

Skill delivery is harness plumbing; skill adoption is a model disposition. Hold the tasks, environments, and verifiers fixed, and goose reproduces the SkillsBench effect — provided the extension is on and the model is willing to reach for what it's given. Feedback welcome, especially from the SkillsBench authors if we've misread any aspect of the protocol.