Skip to content

Commit c79226d

Browse files
pcarletonpcarleton
committed
feat: add SEP conformance traceability manifest
Add a `conformance traceability` subcommand that generates src/seps/traceability.json — a per-SEP manifest of which declared requirements have a conformance scenario, consumed by plan.modelcontextprotocol.io to track SEP-2484 progress. Coverage is derived from a real suite run, not a source scan: `traceability --results <dir>` reads the checks.json files a run produced and marks each declared requirement tested (its check ID was emitted) or untested. 'tested' means a scenario emitted the check against the reference SDK — NOT that any SDK passes it (per-SDK pass/ fail stays in tier-check). - pure computeTraceability join (unit-tested); reads results, does not reimplement the runner - top-level source provenance; refuses to write an all-untested manifest from an empty results dir (--allow-empty to override) - .github/workflows/traceability.yml refreshes it via `conformance sdk` against the reference SDK and opens a rolling PR; advisory, not a gate
1 parent 70f7ba0 commit c79226d

9 files changed

Lines changed: 887 additions & 0 deletions

File tree

‎.github/workflows/traceability.yml‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
name: Refresh SEP traceability manifest
2+
3+
# Regenerates src/seps/traceability.json by running the conformance suite against
4+
# the reference SDK and recording which check IDs were emitted, then opens a PR
5+
# with the diff. NOT a PR gate — runs on demand / on a schedule and proposes an
6+
# update for review. plan.modelcontextprotocol.io reads the committed file from
7+
# main.
8+
#
9+
# Depends on the `conformance sdk` subcommand (#277), which clones+builds the SDK
10+
# and runs the client+server suites. The `run` job executes third-party SDK code,
11+
# so it has NO repo write token (read-only perms, persist-credentials: false) and
12+
# only uploads results as an artifact; the separate `propose` job holds the
13+
# write/PR permissions and never executes SDK code.
14+
15+
on:
16+
workflow_dispatch:
17+
inputs:
18+
sdk:
19+
description: 'SDK ref to run against (e.g. typescript-sdk@<sha>)'
20+
default: 'typescript-sdk@main'
21+
schedule:
22+
- cron: '0 6 * * 1' # Weekly, Monday 06:00 UTC.
23+
24+
concurrency:
25+
group: traceability-refresh
26+
cancel-in-progress: true
27+
28+
jobs:
29+
run:
30+
runs-on: ubuntu-latest
31+
permissions:
32+
contents: read
33+
env:
34+
SDK_REF: ${{ inputs.sdk || 'typescript-sdk@main' }}
35+
steps:
36+
- uses: actions/checkout@v6
37+
with:
38+
persist-credentials: false # no git token while SDK code runs
39+
40+
- uses: actions/setup-node@v6
41+
with:
42+
node-version: 24
43+
cache: npm
44+
45+
- run: npm ci
46+
- run: npm run build
47+
48+
- name: Run conformance suites against the reference SDK
49+
run: node dist/index.js sdk "$SDK_REF" --mode both --suite all -o results
50+
51+
- name: Fail if no results were produced
52+
run: |
53+
if [ -z "$(find results -name checks.json -print -quit 2>/dev/null)" ]; then
54+
echo "No checks.json produced — the suite run failed; not proposing a manifest."
55+
exit 1
56+
fi
57+
58+
- uses: actions/upload-artifact@v4
59+
with:
60+
name: conformance-results
61+
path: results
62+
retention-days: 7
63+
64+
propose:
65+
needs: run
66+
runs-on: ubuntu-latest
67+
# Requires the repo/org setting "Allow GitHub Actions to create and approve
68+
# pull requests" to be enabled, otherwise `gh pr create` fails.
69+
permissions:
70+
contents: write
71+
pull-requests: write
72+
env:
73+
SDK_REF: ${{ inputs.sdk || 'typescript-sdk@main' }}
74+
steps:
75+
- uses: actions/checkout@v6
76+
- uses: actions/setup-node@v6
77+
with:
78+
node-version: 24
79+
cache: npm
80+
- run: npm ci
81+
- run: npm run build
82+
83+
- uses: actions/download-artifact@v4
84+
with:
85+
name: conformance-results
86+
path: results
87+
88+
- name: Regenerate manifest
89+
run: |
90+
set -euo pipefail
91+
# Record the resolved sha (stable per SDK commit) so the manifest's
92+
# `source` only changes when the SDK actually advances — no per-run noise.
93+
ref="${SDK_REF#*@}"
94+
sha="$(git ls-remote https://github.com/modelcontextprotocol/typescript-sdk.git "$ref" | cut -f1)"
95+
node dist/index.js traceability --results results \
96+
--source "typescript-sdk@${sha:0:12}"
97+
98+
- name: Open/update the rolling refresh PR
99+
env:
100+
GH_TOKEN: ${{ github.token }}
101+
run: |
102+
set -euo pipefail
103+
if git diff --quiet -- src/seps/traceability.json; then
104+
echo "traceability.json unchanged"
105+
exit 0
106+
fi
107+
# One rolling branch/PR, force-updated each run, so the schedule does
108+
# not accrue a new PR every week.
109+
branch="traceability-refresh"
110+
git config user.name 'github-actions[bot]'
111+
git config user.email 'github-actions[bot]@users.noreply.github.com'
112+
git checkout -B "$branch"
113+
git add src/seps/traceability.json
114+
git commit -m "chore: refresh SEP traceability manifest ($SDK_REF)"
115+
git push --force origin "$branch"
116+
gh pr view "$branch" >/dev/null 2>&1 || gh pr create \
117+
--head "$branch" \
118+
--title 'chore: refresh SEP traceability manifest' \
119+
--body 'Automated refresh from a conformance run against the reference SDK. Review the coverage diff before merging.'

‎.prettierignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Generated by `conformance traceability` — formatting is owned by the
2+
# generator (deterministic JSON.stringify), not Prettier. Without this, the
3+
# repo's `prettier --check .` would reformat the file and fight the generator's
4+
# output (and the refresh workflow's `git diff` check).
5+
src/seps/traceability.json

‎AGENTS.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,25 @@ npx @modelcontextprotocol/conformance new-sep <NNNN>
8181

8282
The command looks up PR #`<NNNN>` in `modelcontextprotocol/modelcontextprotocol` (SEP numbers are PR numbers), derives `spec_url` from the `docs/specification/draft/*.mdx` file it changes, and writes `src/seps/sep-<NNNN>.yaml` with TODO `requirements[]` rows. Use `--spec-path` or `--spec-url` to skip the lookup. The `new-sep` Claude Code skill drives the same flow end-to-end, parses the spec diff, and fills in the requirement rows.
8383

84+
### Traceability manifest
85+
86+
`src/seps/traceability.json` is a generated map of, per SEP, which declared `check:` IDs are actually emitted when the conformance suite runs against the reference SDK. It is consumed by plan.modelcontextprotocol.io to track SEP-2484 progress.
87+
88+
The emitted check IDs come from a real suite run (not a source scan), so dynamic (template-literal) IDs resolve to their concrete values. Generate the manifest from a results directory:
89+
90+
```sh
91+
# 1. Run the suite against the reference SDK, collecting checks.json files:
92+
node dist/index.js client --command '<sdk conformance client>' --suite all -o results
93+
node dist/index.js server --url '<sdk conformance server url>' --suite all -o results
94+
# 2. Build the manifest from those results:
95+
npm run traceability -- --results results
96+
npm run traceability -- --results results --strict # exit 1 on any untested (advisory)
97+
```
98+
99+
Each requirement is `tested` (its check ID was emitted) or `untested` (declared but never emitted — a real gap, or a check that only fires against a deliberately-broken impl, i.e. it needs a negative test). `"tested" means a scenario emitted the check ID, NOT that any SDK passes it` — per-SDK results live in `tier-check`. Matching is exact, so a scenario's emitted check IDs must match the requirement slugs in the yaml (one check ID per MUST/SHOULD, emitted once per case).
100+
101+
The manifest is refreshed by `.github/workflows/traceability.yml` (manual/scheduled), which runs the suite against typescript-sdk and opens a PR with the diff — it is **not** a PR gate. Untested checks are advisory for now; the intended future policy is that an untested check must be backed by a negative test.
102+
84103
## Examples: prove it passes and fails
85104

86105
A new scenario should come with:
@@ -100,3 +119,4 @@ Use the existing CLI runner (`npx @modelcontextprotocol/conformance client|serve
100119
- `npm test` passes
101120
- For non-trivial scenario changes, run against at least one real SDK (typescript-sdk or python-sdk) to see actual output. For changes to shared infrastructure (runner, tier-check), test against go-sdk or csharp-sdk too.
102121
- Scenario is registered in the right suite in `src/scenarios/index.ts`
122+
- If you changed a `sep-*.yaml` or scenario check IDs, `src/seps/traceability.json` will drift; the traceability workflow refreshes it via PR (or regenerate locally with `--results` from a suite run)

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
"lint:fix": "eslint src/ examples/ --fix && prettier --write .",
2020
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet",
2121
"tier-check": "node dist/index.js tier-check",
22+
"traceability": "tsx src/index.ts traceability",
2223
"check": "npm run typecheck && npm run lint",
2324
"typecheck": "tsgo --noEmit",
2425
"prepack": "npm run build",

‎src/index.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ import {
4646
} from './expected-failures';
4747
import { createTierCheckCommand } from './tier-check';
4848
import { createNewSepCommand } from './new-sep';
49+
import { createTraceabilityCommand } from './traceability';
4950
import packageJson from '../package.json';
5051

5152
// Note on naming: `command` refers to which CLI command is calling this.
@@ -544,6 +545,9 @@ program.addCommand(createTierCheckCommand());
544545
// New SEP scaffolding command
545546
program.addCommand(createNewSepCommand());
546547

548+
// SEP traceability manifest command
549+
program.addCommand(createTraceabilityCommand());
550+
547551
// List scenarios command
548552
program
549553
.command('list')

‎src/seps/traceability.json‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
{
2+
"schemaVersion": 1,
3+
"meaning": "Per-SEP presence of conformance scenarios for each declared requirement. \"tested\" means the check ID was emitted when the conformance suite ran against the reference SDK — NOT that any SDK passes it. Per-SDK pass/fail lives in tier-check (not yet joinable: tier-check does not currently expose per-check IDs).",
4+
"contract": "A SEP appears here only if it has a traceability yaml or emits sep-NNNN-* check IDs at runtime. A SEP absent from this file has no conformance artifacts (treat as not-started). Emitted IDs come from running the suite against the reference SDK (positive paths). \"untested\" therefore folds together a real coverage gap, a check that only fires against a broken impl (needs a negative test), and a check the reference SDK gates off because it has not implemented the feature yet. The signal is advisory.",
5+
"source": "typescript-sdk@6f0bf49d",
6+
"seps": {
7+
"2164": {
8+
"yaml": "src/seps/sep-2164.yaml",
9+
"specUrl": "https://modelcontextprotocol.io/specification/draft/server/resources#error-handling",
10+
"requirements": [
11+
{
12+
"check": "sep-2164-no-empty-contents",
13+
"status": "tested",
14+
"text": "Servers MUST NOT return an empty contents array for a non-existent resource"
15+
},
16+
{
17+
"check": "sep-2164-error-code",
18+
"status": "tested",
19+
"text": "Servers SHOULD return standard JSON-RPC errors for common failure cases: Resource not found: -32602 (Invalid Params)"
20+
}
21+
],
22+
"excluded": [
23+
{
24+
"text": "clients SHOULD also accept -32002 as a resource not found error",
25+
"reason": "Client-side error handling is implementation-defined; not protocol-observable"
26+
}
27+
],
28+
"unkeyed": [],
29+
"untracked": [
30+
"sep-2164-data-uri"
31+
],
32+
"summary": {
33+
"tested": 2,
34+
"untested": 0,
35+
"excluded": 1,
36+
"untracked": 1,
37+
"unkeyed": 0
38+
}
39+
},
40+
"2207": {
41+
"yaml": null,
42+
"specUrl": null,
43+
"requirements": [],
44+
"excluded": [],
45+
"unkeyed": [],
46+
"untracked": [
47+
"sep-2207-client-metadata-grant-types",
48+
"sep-2207-offline-access-not-requested",
49+
"sep-2207-offline-access-requested"
50+
],
51+
"summary": {
52+
"tested": 0,
53+
"untested": 0,
54+
"excluded": 0,
55+
"untracked": 3,
56+
"unkeyed": 0
57+
}
58+
},
59+
"2243": {
60+
"yaml": "src/seps/sep-2243.yaml",
61+
"specUrl": "https://modelcontextprotocol.io/specification/draft/basic/transports#standard-mcp-request-headers",
62+
"requirements": [
63+
{
64+
"check": "sep-2243-client-includes-standard-headers",
65+
"status": "tested",
66+
"text": "The client MUST include the standard MCP request headers on each POST request. These headers are REQUIRED for compliance."
67+
},
68+
{
69+
"check": "sep-2243-header-name-case-insensitive",
70+
"status": "tested",
71+
"text": "Clients and servers MUST use case-insensitive comparisons for header names."
72+
},
73+
{
74+
"check": "sep-2243-server-reject-invalid-headers",
75+
"status": "tested",
76+
"text": "Servers that process the request body MUST reject requests with mismatched or missing standard-header values, returning HTTP 400 Bad Request."
77+
},
78+
{
79+
"check": "sep-2243-server-reject-error-code",
80+
"status": "tested",
81+
"text": "When rejecting a request due to header validation failure, servers SHOULD include a JSON-RPC error response using error code -32001."
82+
},
83+
{
84+
"check": "sep-2243-client-supports-custom-headers",
85+
"status": "untested",
86+
"text": "MCP clients MUST support this feature [custom headers via x-mcp-header]."
87+
},
88+
{
89+
"check": "sep-2243-client-mirrors-designated-params",
90+
"status": "untested",
91+
"text": "When a client invokes a tool whose definition includes such designations, conforming clients MUST mirror the designated parameter values into HTTP headers as described below."
92+
},
93+
{
94+
"check": "sep-2243-x-mcp-header-not-empty",
95+
"status": "untested",
96+
"text": "The x-mcp-header value MUST NOT be empty.",
97+
"url": "https://modelcontextprotocol.io/specification/draft/server/tools#custom-headers"
98+
},
99+
{
100+
"check": "sep-2243-x-mcp-header-charset",
101+
"status": "untested",
102+
"text": "The x-mcp-header value MUST contain only ASCII characters (excluding space and `:`).",
103+
"url": "https://modelcontextprotocol.io/specification/draft/server/tools#custom-headers"
104+
},
105+
{
106+
"check": "sep-2243-x-mcp-header-unique",
107+
"status": "untested",
108+
"text": "The x-mcp-header value MUST be case-insensitively unique within a single tool definition.",
109+
"url": "https://modelcontextprotocol.io/specification/draft/server/tools#custom-headers"
110+
},
111+
{
112+
"check": "sep-2243-x-mcp-header-primitive-only",
113+
"status": "untested",
114+
"text": "x-mcp-header MUST only be applied to parameters with primitive types (number, string, or boolean).",
115+
"url": "https://modelcontextprotocol.io/specification/draft/server/tools#custom-headers"
116+
},
117+
{
118+
"check": "sep-2243-client-reject-invalid-tool",
119+
"status": "tested",
120+
"text": "Clients MUST reject tool definitions where any x-mcp-header value violates these constraints. Rejection means the client MUST exclude the invalid tool from the set of tools returned by tools/list.",
121+
"url": "https://modelcontextprotocol.io/specification/draft/server/tools#custom-headers"
122+
},
123+
{
124+
"check": "sep-2243-client-encode-values",
125+
"status": "untested",
126+
"text": "Clients MUST encode parameter values before including them in HTTP headers: number values MUST be converted to their decimal string representation; boolean values MUST be converted to the lowercase strings \"true\" or \"false\"."
127+
},
128+
{
129+
"check": "sep-2243-client-base64-unsafe",
130+
"status": "untested",
131+
"text": "When a value cannot be safely represented as plain ASCII (e.g., contains non-ASCII characters, control characters, or leading/trailing whitespace), clients MUST use Base64 encoding of the UTF-8 representation, wrapped as =?base64?{encoded}?=."
132+
},
133+
{
134+
"check": "sep-2243-server-decode-base64",
135+
"status": "untested",
136+
"text": "Servers and intermediaries that need to inspect these values MUST decode them accordingly."
137+
},
138+
{
139+
"check": "sep-2243-client-omit-null",
140+
"status": "tested",
141+
"text": "Parameter value is null or omitted: Client MUST omit the header."
142+
},
143+
{
144+
"check": "sep-2243-server-not-expect-null",
145+
"status": "untested",
146+
"text": "Parameter value is null or omitted: Server MUST NOT expect the header."
147+
},
148+
{
149+
"check": "sep-2243-server-reject-missing-required",
150+
"status": "untested",
151+
"text": "Required parameter is omitted: Server MUST reject with JSON-RPC error."
152+
},
153+
{
154+
"check": "sep-2243-server-reject-invalid-param-chars",
155+
"status": "untested",
156+
"text": "Servers MUST reject requests with a recognized Mcp-Param-{Name} header that contain invalid characters."
157+
},
158+
{
159+
"check": "sep-2243-server-validate-param-match",
160+
"status": "untested",
161+
"text": "Any server that processes the message body MUST validate that encoded header values, after decoding if Base64-encoded, match the corresponding parameter values in the body."
162+
},
163+
{
164+
"check": "sep-2243-server-reject-param-mismatch",
165+
"status": "untested",
166+
"text": "Servers MUST reject requests with a 400 Bad Request HTTP status and JSON-RPC error code -32001 if any validation fails."
167+
}
168+
],
169+
"excluded": [
170+
{
171+
"text": "Clients SHOULD log a warning when rejecting a tool definition due to invalid x-mcp-header, including the tool name and the reason.",
172+
"reason": "Log output is not wire-observable."
173+
},
174+
{
175+
"text": "Server developers SHOULD NOT mark sensitive parameters (such as passwords, API keys, tokens, or PII) with x-mcp-header.",
176+
"reason": "Design guidance to humans; not protocol-observable."
177+
},
178+
{
179+
"text": "Intermediaries MUST return an appropriate HTTP error status for validation failures.",
180+
"reason": "Intermediary requirement; conformance harness tests clients and servers, not intermediaries."
181+
},
182+
{
183+
"text": "Intermediate servers that do not recognize an Mcp-Param-{Name} header MUST forward it and otherwise ignore it.",
184+
"reason": "Intermediary requirement; conformance harness tests clients and servers, not intermediaries."
185+
}
186+
],
187+
"unkeyed": [],
188+
"untracked": [
189+
"sep-2243-invalid-tool-tools-list-gate",
190+
"sep-2243-param-header-tool-call-gate",
191+
"sep-2243-server-accepts-whitespace-header-value",
192+
"sep-2243-server-no-xmcp-tool"
193+
],
194+
"summary": {
195+
"tested": 6,
196+
"untested": 14,
197+
"excluded": 4,
198+
"untracked": 4,
199+
"unkeyed": 0
200+
}
201+
}
202+
}
203+
}

0 commit comments

Comments
 (0)