Compare skill delivery mechanisms: file-based vs MCP-based

[olaservo]

Background

The group has discussed whether models reliably discover and follow skills delivered via MCP compared to file-based approaches (e.g., agent pointed to a skills folder). While general model reliability is the responsibility of model providers and client harnesses, this group should verify that MCP-based skill delivery works at least as well as file-based delivery — and identify any gaps.

Cross-provider model-level reliability testing is already underway externally in skilljack-evals.

Scope

Compare skill delivery mechanisms to ensure MCP-based delivery is competitive with file-based approaches:

1. File-based (baseline): Agent pointed to a folder containing SKILL.md files — the default approach for most clients today
2. MCP-based (under test): Skills delivered as MCP resources, with variations:
   • Resource-only (skill available as a resource for the agent to read)
   • Resource + server instructions (server instruction pointing agent to the skill resource)
   • Tool-description-embedded (skilljack pattern — skill content in tool descriptions)

Key questions

• Does MCP-based delivery result in comparable skill discovery and adherence vs file-based?
• Which MCP delivery pattern (resources, server instructions, tool descriptions) is most reliable?
• Are there gaps in MCP that make skill delivery less effective than file-based?

Acceptance Criteria

• Documented test methodology comparing file-based vs MCP-based delivery
• Results for at least 2 MCP delivery patterns vs file-based baseline
• Identified gaps (if any) where MCP delivery is less reliable
• Recommendations for how MCP should deliver skills to align with existing patterns
• Findings added to experimental-findings.md

References

• Skill Reliability and Adherence in experimental-findings.md
• skilljack-evals — Cross-provider model-level skill adherence testing
• skilljack-mcp evals
• Open Question 5

[LucaButBoring]

I would suggest at least two models from different providers, specifically - I'm concerned having all tested models be from the same provider will bias heavily in favor of however that provider tends to design their training regimens.

[LucaButBoring]

I suppose that gets difficult if you plan to test with the Claude Agent SDK specifically, we could maybe proxy to OpenRouter or something with ccproxy? That's a bit uncontrolled in its own right because it'll still have the Claude system prompt, but it's better than nothing if we can't test with different harnesses.

[olaservo]

@LucaButBoring for simple file-based/non-MCP skill usage, I've been experimenting with other providers and models here: https://github.com/olaservo/skilljack-evals/tree/main

This repo produced some interesting results so far between Claude, GPT, and a collection of open-source models with a very simple 'hello world' style skill: https://github.com/olaservo/skilljack-evals/actions/runs/22659359058 (tldr; they almost all pass my threshold for getting invoked correctly by the model, but GPT 5.2 consistently fails my 'false positive' test where it shouldn't be invoked.)

I was thinking of adapting this approach to also support testing skills over MCP. Even though the most useful 'real world' experiments should probably come from testing the readymade agents that support both file-based skill discovery and skills as MCP resources, testing with different models does seem useful to reveal any model-specific behavioral differences.

[pja-ant]

Is this an orthogonal concern to the work of this group? I feel like its the harness's + model's job to do better at this. Especially since skills are so new, I'd expect this to get dramatically better as providers tune their models around it and models get better. IMO the best thing we can do from MCP standpoint is to try and deliver skills in a way that is most aligned with how existing skills are delivered so that we ride that wave of improvement.

[olaservo]

I feel like its the harness's + model's job to do better at this. Especially since skills are so new, I'd expect this to get dramatically better as providers tune their models around it and models get better. IMO the best thing we can do from MCP standpoint is to try and deliver skills in a way that is most aligned with how existing skills are delivered so that we ride that wave of improvement.

I think that generally makes sense. The main reason this came up in this group is because 'not being invoked or followed' was raised as a general concern with Resources being less reliable than other methods for exposing skills to agents.

In those cases, I suspect the main gap is not having a reliable mechanism for the client application to expose the skill resources (including skill descriptions) to the model.

Now that we're prototyping a recommended pattern to solve for that, I think it still makes sense to compare 'pointing an agent to a folder' (i.e. using the default approach for clients today) vs 'providing an agent with a way to discover and use skills through MCP,' (since different methods couuld still show variations in how they decide to select and use the SKILL.md and files)?

[olaservo]

I edited the issue to narrow the scope since I agree that 'how well do given models + harnesses use skills' is largely the responsibility of model providers and harness developers. That said, I don't think its 100% orthogonal. If models can't reliably use skills delivered via MCP, we need to know whether that's a model problem or a delivery problem. The updated scope focuses on what we can control (i.e. comparing MCP-based delivery against the file-based baseline).