validation.md

Validation

Every check in the repository, what it verifies, and how to fix a failure. All checks are offline and deterministic — none require the network, API tokens, or model-provider keys.

Run the full suite before committing or cutting a release.

Quick reference

node scripts/sync-artifacts.mjs --check       # artifact drift + version metadata
pnpm test:sync                                # sync tool unit tests
node scripts/release-assets.mjs --check-manifest       # release asset manifest shape
node scripts/release-assets.mjs --check-install-json   # install.json vs latest published
node scripts/release-assets.mjs --check-local-packages # package source paths + files
pnpm test:release-assets                      # release/distribution unit tests
node scripts/check-npm-package.mjs            # npm package readiness (no publish)
bash scripts/verify-packages.sh               # pack/skill completeness
bash scripts/evaluate-skills.sh               # skill evaluation suite
pnpm eval:skills                              # same suite via package script
bash scripts/validate-agent-files.sh          # agent/pack/docs structure + content
bash scripts/validate-docs.sh                 # docs surface + version pointers
bash scripts/validate-release.sh              # version parity + release readiness
bash scripts/validate-skill.sh                # skill package + eval scaffold
bash scripts/validate-bilingual.sh            # bilingual examples/templates
bash scripts/test-install-codex.sh            # Codex installer behavior
pnpm --filter @oh-my-pm/mcp-server test       # MCP server unit tests
pnpm --filter @oh-my-pm/cli test              # CLI unit tests

If pnpm is unavailable, use npx --yes pnpm@8 in its place.

Per-command detail

node scripts/sync-artifacts.mjs --check

• Checks: mirrored artifacts are byte-identical to their canonical source, and every source-version location equals the manifest version.
• When: before every commit and in validate-agent-files.sh / validate-release.sh.
• Common failures: you edited a mirror target directly, or a version location is out of date.
• Fix: edit the canonical source, then node scripts/sync-artifacts.mjs --write (--prune to remove stale targets). For version drift, update source metadata.
• Network/credentials: none.

pnpm test:sync

• Checks: the sync tool's core logic (drift detection, write restoration, stale handling, excludes, write-boundary safety, version extraction) via temp-directory fixtures.
• When: after changing the sync tool or manifest.
• Common failures: a regression in scripts/sync-lib.mjs.
• Fix: address the failing assertion.
• Network/credentials: none.

bash scripts/verify-packages.sh

• Checks: every pack/skill root, VERSION, and README exists; mirrors are in sync; no stray build artifacts inside packaged trees.
• When: before packaging a release.
• Common failures: a missing pack file or artifact drift.
• Fix: restore the missing file or run the sync tool.
• Network/credentials: none.

bash scripts/evaluate-skills.sh / pnpm eval:skills

• Checks: the offline skill evaluation suite (scenarios vs. golden outputs, scored against the rubric).
• When: after changing skill content or evaluation fixtures.
• Common failures: a scenario's output no longer matches the golden/rubric.
• Fix: update the skill content or the golden output intentionally.
• Network/credentials: none. Does not call a live model — it is a deterministic offline check, not a live-quality judge.

bash scripts/validate-agent-files.sh

• Checks: required agent/pack/docs files exist and contain required content; MCP read-only/health surfaces are documented; connector parity; artifact sync passes.
• When: before every commit.
• Common failures: a missing/renamed file, or a required doc phrase absent.
• Fix: address the named check.
• Network/credentials: none.

bash scripts/validate-release.sh

• Checks: version parity across all source-version locations; install.json and README badge point only to the latest published version; dist scaffolding; the eval suite; artifact sync; no staged zips/dist.
• When: before tagging/publishing.
• Common failures: version metadata drift, or install.json pointing at the wrong published version.
• Fix: align source metadata to EXPECTED_VERSION; keep install.json at LATEST_PUBLISHED_VERSION until the post-release freshness patch.
• Network/credentials: none.

bash scripts/validate-skill.sh

• Checks: skill package files and the evaluation scaffold exist; no forbidden authorship strings; golden outputs make no first-person write claims.
• When: after changing skill or eval files.
• Fix: address the named check.
• Network/credentials: none.

bash scripts/validate-bilingual.sh

• Checks: paired EN/FA examples and templates exist and are well-formed.
• When: after changing bilingual content.
• Fix: add the missing counterpart or correct formatting.
• Network/credentials: none.

bash scripts/test-install-codex.sh

• Checks: the Codex installer's behavior (install, overwrite guard, backup, dry-run, version output).
• When: after changing the Codex installer.
• Fix: address the failing assertion.
• Network/credentials: none.

pnpm --filter @oh-my-pm/mcp-server test

• Checks: MCP server unit tests — formatting, config, read-only policy, health model, capability parity, read-only boundary, mock connectors.
• When: after changing MCP server code.
• Fix: address the failing test.
• Network/credentials: none. Tests use fixtures/mocks and scrub env vars — they never call live APIs.

pnpm --filter @oh-my-pm/cli test

• Checks: CLI unit tests — argument parsing, install/skill/mcp/doctor commands, secret-safe doctor output.
• When: after changing CLI code.
• Fix: address the failing test.
• Network/credentials: none.

Parity checks

diff -rq codex-skill/oh-my-pm packs/codex/.agents/skills/oh-my-pm
diff -rq .cursor/rules packs/cursor/.cursor/rules

Both must report no differences. If they do, run the artifact sync tool.

Related

• artifact-sync.md — the sync/drift system
• releases.md — how validation gates a release
• evaluations.md — the skill evaluation system in depth