Why Agents Need Skills: Doing the Job the Way You Want, Unsupervised

Introduction

We kept hitting the same wall. You can wire an agent up to every tool it needs, point it at a real workflow, and let it run—and it will do a job. Just not always your job. It skips the approval step your compliance team insists on. It refunds a customer who was outside the policy window. It gets a plausible-looking outcome by a path nobody signed off on. Nobody was watching, so nobody caught it. That is the difference between an agent that is automatic and one that is autonomous, and it is the whole ballgame for operational work.

So we asked a narrow, testable question: what actually makes an agent do the job the way you want, with no one in the loop to correct it? Our hypothesis was that the binding constraint isn't the model's raw capability—it's whether the right Standard Operating Procedure (SOP) reaches the agent's context at the right moment, with no human choosing it. And that the mechanism you use to deliver that SOP—a skill, a tool, or an MCP prompt—is what decides the outcome.

We tested it. Three benchmarks, deterministic oracles, five arms from no SOP at all to an auto-discovered SOP with externalized run-tracking, all on claude-haiku-4-5. Here's the short version of what we found: giving the agent the SOP raises single-turn task success by +0.35 (95% CI [+0.26, +0.45], p<0.001) and pushes multi-turn procedure conformance from ~50–59% up to ~90%. The delivery channel—skill in context versus tool retrieval—is a wash. And bolting heavyweight run-tracking on top buys nothing but a 2–4× tool-call bill.

What's inside:

• Why "autonomous" has to mean conforms and succeeds unsupervised—not merely "runs unsupervised"
• The one property a skill has that an MCP prompt structurally cannot—auto-discovery—and why it decides everything
• The numbers: what supplying the SOP buys, and the two negative results that save you money
• An honest read on how much to trust each claim—this is an engineering study, not a peer-reviewed benchmark

One thing up front, because it shapes how you should read the rest: this study was sized for budget and compute, not for statistical power. The single-turn success gain and the conformance jump on τ² are solid. The multi-turn magnitudes are directional. We'll flag which is which as we go, and there's a whole section on it before the numbers.

What autonomy actually requires

Start with a definition, because the loose one causes all the trouble. For an agent to do operational work on its own, four things have to hold, and they're exactly what we set out to measure.

Conformance is whether the agent does the job the way you want—it follows the approved SOP, the right steps in the right order, honoring the constraints (the headline metric, measured first). Success is whether it produces the outcome you want: the correct result. Autonomy is achieving both, with no person in the loop. That last word is stricter than it sounds. An agent that merely runs unattended but conforms poorly, or lands on a different outcome, is not autonomous—it's only automatic, and you cannot trust it alone. Real autonomy requires the agent to obtain and apply the right SOP by itself.

And then there is efficiency—doing all of that without wasteful overhead, in a way that keeps scaling as your SOP library grows. This is not a footnote; it is the fourth thing we measure, and it is where the delivery mechanism earns its keep. Auto-discovery keeps only the skill descriptions resident and loads a full procedure into context only when it matches, rather than carrying every SOP at once. A benchmark that ignored efficiency would miss the difference between a design that works with ten SOPs and one that still works with a thousand—and, as the tool-call counts later show, between a lean skill and a heavyweight run-tracking rig that buys nothing.

The binding constraint is autonomy. The agent can only conform and succeed on its own if the right SOP reaches its context at the right moment, with no human choosing it. That is where the delivery mechanism decides everything—and where the difference between a skill and an MCP prompt stops being academic.

A skill carries auto-discovery. The runtime surfaces each skill's description; when a task arrives, the agent itself matches it to the relevant skill and loads the full procedure on demand. No human picks it. Standing context stays small—only the descriptions are resident until something matches—so the approach scales as the library grows. An MCP prompt has no auto-discovery. A prompt is a user-invoked template that a person selects from a menu. With no user present, the procedure never enters context and the agent improvises. That's the failure mode in one sentence: automatic, not autonomous.

This isn't only our framing. It's why there's an active working group to add skills to the Model Context Protocol—tools and prompts alone don't let an agent discover and apply the correct procedure by itself. A tool can carry a procedure, sure. But nothing about a tool or a prompt makes the agent reach for the right one unprompted. That gap is the reason skills exist as a distinct primitive.

The whole exercise reduces to one claim: without a discoverable, self-applied SOP, the agent does neither its job the right way nor reaches the right outcome. With one delivered as a skill, it does both. So we measured conformance and success for every arm—including the ones where no SOP is supplied at all.

How we tested it

We built a ladder of five arms, climbing from no procedure to an auto-discovered procedure with run-tracking. Each arm is one agent per domain, scoped to that domain's tools, graded by the benchmark's own deterministic oracle.

The skills arm is delivered as a filesystem SKILL.md the runtime auto-discovers—the same mechanism a production agent would use. The two planning arms retrieve a hand-authored, completeness-checked stepwise procedure through an in-process MCP tool. Critically, the procedure text is identical across both planning arms and is track-agnostic, so the planning-tracking − planning contrast isolates the run-tracking protocol alone—nothing else moves.

We ran this across three public benchmarks, each with a deterministic oracle that checks the agent's actual behavior rather than asking a model to grade it. They span single-turn and multi-turn, with a simulated user driving the conversation in the multi-turn cases.

Scope, sampling, and confidence—read this before the numbers

This is an engineering study, not a peer-reviewed benchmark, and the design was sized for budget and compute. Treat the numbers as directional evidence, not definitive measurement. Four things constrain what we can claim, and we'd rather you hear them from us than infer them later.

We subsampled hard. Each benchmark ships far more tasks than we ran—SOP-Bench has ~2,154, SOPBench ~830, τ² ~278. We evaluated a deterministic 10-tasks-per-domain subsample (τ² at 15/domain), which is 45–140 runs per arm. Multi-turn cells cost roughly 50–200K tokens each, because every task runs an agent plus an LLM user-simulator loop, and they share an org rate limit. A full-pool × 5-arm × repeats sweep simply wasn't affordable. Ten per domain is a deliberate coverage/cost trade-off, not a power calculation.

We ran each cell once (R=1). Cells weren't repeated, so cross-run reliability (pass^k) is untested, and each point estimate carries run-to-run stochastic noise on top of sampling noise. One model tier. Everything is claude-haiku-4-5—no cross-model comparison. The absolute effects, and especially the externalized-state trade-offs, should move with stronger models.

The statistics are paired. Every contrast is computed within (benchmark, domain, task), and we report paired-bootstrap 95% CIs (≥10k resamples) plus McNemar's test on discordant pairs. We only call a result significant when the CI excludes zero. At these sample sizes, single-task differences below ~10 percentage points sit inside the noise. When we quote a per-arm mean gap that looks bigger than the paired contrast, trust the paired number—it's the honest one.

The headline: does the agent do the job the way you want?

This is the metric the whole study exists for. Conformance is the fraction of the prescribed procedure the agent actually follows—higher means closer to the SOP.

Look at τ². The skill arm follows roughly 90% of the prescribed policy actions, against ~59% with domain context only and ~50% with nothing. That's a paired, statistically significant conformance gain of +0.16 over baseline (CI [+0.06, +0.28]) and +0.28 over null (CI [+0.16, +0.40]). The raw per-arm-mean gap looks larger—closer to +0.32/+0.40—because the arms don't run a byte-identical task set. The paired contrast, computed the same way as our success contrasts, is roughly half that, and it's the number to quote.

On SOPBench the direction is identical but the effect is small and not significant (paired +0.04, CI [−0.03, +0.11]; most tasks tie). So the honest summary is: giving the agent the SOP measurably improves how faithfully it follows the procedure—clearly on τ², inconclusively on SOPBench—and it never degrades conformance anywhere.

Conformance also means not acting when you shouldn't. SOPBench labels each task as should-act or should-decline, and the SOP improves both sides. Correct declines on should-decline tasks rise from 0.13 (null) to 0.26 (skills), and execution on should-act tasks from 0.03 to 0.06. The absolute rates are low and the subsets small (n≈32 act / 38 decline, SOPBench only), so read this as directional. But it points the right way: an agent given the SOP is better both at doing the job and at correctly refusing out-of-policy requests. That second half is exactly what you need before you trust it unattended.

† SOP-Bench is single-turn and has no action-sequence oracle, so its only conformance proxy is tool coverage—did the agent call the prescribed domain tools. Single-turn tasks force those calls regardless of the SOP, so the proxy is uninformative here; it even reads backward (baseline 0.75 > skills 0.52, because the skill sometimes decides with fewer tool calls). Real conformance is measured on the multi-turn benchmarks, whose oracles actually check whether the agent followed the procedure—and there the SOP clearly makes the agent conform.

The outcome: does it reach the result you want?

Task success rate (TSR) is the benchmark's native measure of whether the agent produced the correct final result.

The key contrast is skills − baseline = +0.35 on SOP-Bench (CI [+0.26, +0.45], p<0.001, significant), +0.04 on SOPBench (not significant), and +0.16 on τ² (not significant). Read that as: supplying the SOP drives the outcome too—large and significant in the clean single-turn regime, directional under the noisier multi-turn one. Conformance and success move together. With the skill, the agent does the job the right way and lands on the right answer—the definition of autonomous. Without it, it reliably does neither: merely automatic.

It's worth being clear-eyed about the multi-turn magnitudes. The τ² success lift is real in direction but its size is inside the noise band at n=45. We'd need more tasks per domain, repeats, and a second model tier to move that from "directional" to "measured." The single-turn success gain, by contrast, is one of the two claims we'd stand behind without hedging.

Two negative results that save you money

The most useful findings here are the ones that tell you what not to build.

The delivery channel is a wash

Compare planning (SOP pulled as a tool) against skills (SOP auto-discovered in context): +0.03 on SOP-Bench, −0.01 on SOPBench, −0.04 on τ²—every one inside the noise, on both conformance and success. Once the procedure is available to the agent, the channel shows no detectable difference in outcome at this power. So the case for skills is not higher scores. It's discoverability: the agent selecting and applying the right procedure with no human invocation. That is the property a skill has and an MCP prompt does not. We didn't run a user-invoked prompt arm—because in an unsupervised run there is no user to invoke it, which is exactly the point.

Externalized run-tracking is net-negative

Now the cost side. Here are the mean tool calls per run—lower is leaner.

The auto-discovered skill is the leanest delivery: it applies the procedure with the fewest tool calls and keeps standing context small, since only descriptions are resident until a skill matches. Bolting an externalized run-tracking protocol on top—the planning-tracking arm—buys no additional conformance or success (the planning-tracking − planning success contrast is −0.06 / +0.01 / +0.09, all not significant) while costing ~2–4× the tool calls. Discovery plus on-demand delivery is the efficient design. Heavyweight state machinery is pure overhead at these horizons.

The caveat worth naming: our tasks are short, 4–15 steps. The strongest theoretical case for externalized tracking is the long composite session where in-context self-tracking degrades—and that regime is untested here. If your workflows are genuinely long-horizon, don't read this as "never track." Read it as "don't pay for tracking you can't yet show you need."

Automatic is not autonomous

One result is tempting to misread, so let's handle it head-on. On SOPBench, the SOP cuts the follow-up turns a user has to supply from ~6 (no SOP) down to ~0.5–1.8. The agent runs with far less hand-holding. That's nice—but it is not the goal, and it is not the definition of autonomy.

An agent that needs zero human turns yet does the job the wrong way, or gets the wrong answer, is only automatic. Autonomy is conformance and success: the right way and the right outcome. Fewer interruptions is a side-effect, not the objective. τ² is the reminder that keeps you honest here—its tasks require eliciting information from the user, so the more capable arms actually take more turns. If you optimized for "fewer turns," you'd wrongly flag the best agents as the worst ones.

What this means for how you build agents

Pulling it together, four things follow. First, autonomy is "does the job your way and gets your outcome, unsupervised"—not merely "runs unsupervised." The two metrics that define it are conformance and success, and you should be measuring both, not uptime or turn count.

Second, the SOP is what delivers both. Without it, the agent under-conforms and under-succeeds—automatic at best. With the SOP as a skill, conformance improves significantly on τ² (the skill arm follows ~90% of prescribed actions versus ~50–59% without) and success rises with it, with the honest caveat that the SOPBench conformance gain is small and the multi-turn success magnitudes are directional.

Third, the value of a skill is auto-discovery, not a score edge. The agent has to select and apply the right SOP with no user present. Skills do that; user-invoked MCP prompts structurally cannot—which is why skills belong in the agent's toolkit, and why the MCP working group is adding them. Fourth, keep it lean. Discovery plus on-demand delivery is the win. Externalized run-tracking is cost without benefit at these horizons, so don't reach for it until your sessions are long enough to justify it.

So the "do this Monday" version: package the procedures that define how your operators actually work as auto-discoverable skills, not as prompts a human has to pick and not as bare tools. Measure your agents on whether they follow those procedures and reach the right outcome—both, unsupervised. And resist the urge to build a run-state machine until you can show the horizon demands one. As for which work an agent can own in the first place and which stays human—that's a question of position in the arc of work, the companion argument in The Tideline.

To make an agent do operational work autonomously—the way you want, with the outcome you want, and no one watching—it must discover and apply the correct SOP by itself. Auto-discovered skills provide exactly that. With a discoverable, self-applied SOP the agent conforms significantly better (τ²) and succeeds more (single-turn); without one it does neither reliably, and is only automatic.

What the MCP spec for skills actually looks like

We've leaned on the phrase "an active working group is adding skills to MCP" a couple of times now. It's worth being concrete about what that group is building, because the spec is converging on precisely the mechanism our skills arm relied on—description-first discovery, full instructions loaded on demand—and pointedly not on the prompts primitive.

The Skills Over MCP working group formed as an interest group in February 2026 and became a full working group that April, co-led by a maintainer from Anthropic and one from Nordstrom, with participants from Google, GitHub, AWS, Databricks, Bloomberg, Saxo Bank and others. It grew out of one sharp question, raised in SEP-2076 ("Agent Skills as a First-Class MCP Primitive"): do MCP's existing primitives—tools, prompts, resources—already suffice for skills, or do skills need a primitive of their own?

Two answers are on the table. SEP-2076 proposes skills as a brand-new first-class primitive, with its own skills/list and skills/get methods, a skills server capability, and a notifications/skills/list_changed signal—the same shape tools and prompts already have. The working group's current direction, though, is the leaner SEP-2640 ("Skills Extension," on the Extensions Track), which builds skills on the existing Resources primitive rather than adding a new one. Reusing Resources inherits URI addressability, resources/read, subscriptions and templates for free; a server declares the capability as io.modelcontextprotocol/skills and can then serve its tools and the instructions for using them as a single atomic unit.

A skill, in the draft, is a directory with a SKILL.md at its root carrying YAML frontmatter—at minimum a name and a description—plus any supporting files, and no nested SKILL.md beneath it. Each skill gets a skill:// URI (for example skill://acme/billing/refunds/SKILL.md), whose final path segment must equal the skill's name. An optional skill://index.json lists what's available, but a skill's URI is directly readable via resources/read whether or not it's indexed.

The important part, for our purposes, is how it's meant to be consumed. The host loads only the frontmatter—name and description—into the model's context at startup; the model then calls resources/read to pull the full instructions only when a task matches. That is progressive disclosure, and it is exactly the property that let the skills arm keep standing context small while still applying the right procedure on demand—the efficiency result the tool-call counts bear out. It is also, pointedly, not how prompts work: a prompt is still something a user picks. Skills get their own discovery path precisely so the agent can route to them with no human in the loop—the autonomy property this whole piece is about.

None of this is final—SEP-2640 and its reference implementation were both still in review as of writing, with related work underway on registry-level discovery (a skills.json proposal) and coordination with the external Agent Skills content spec. But the direction is unambiguous, and it matches what our benchmark points at: give the agent a discoverable, self-applied procedure, and it conforms and succeeds on its own.

What we'd instrument next

Two honest gaps sit at the center of the skills argument, and we're naming them because they're where the next round of evidence has to come from. This experiment shows that the SOP helps and that the delivery channel among agent-driven options is a wash. It does not empirically show skills beat MCP prompts—we ran no user-invoked prompt arm—nor does it measure auto-discovery accuracy. The case for skills specifically rests on the auto-discovery design property, argued, not on a measured score advantage. We'd rather say that plainly than let it be overread.

The highest-value addition is routing / discovery accuracy: when a task arrives, did the agent auto-discover and load the correct SOP from the library, with no user? That measures auto-discovery directly—the property a skill has and a user-invoked prompt lacks—and it isolates a real failure mode of scaling the library, mis-routing. We don't log selected-versus-correct skill today; we should. Second is reliability (pass^k, R≥5). We ran one repeat per cell, so consistency across stochastic runs is untested—and reliability is both the natural place for externalized state to finally pay off and the thing that matters most for unsupervised operation, where a single rare wrong run is expensive.