diff --git a/GOVERNANCE.md b/GOVERNANCE.md index fc643e79..d0e0fe17 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -64,7 +64,7 @@ none object. ## Plugins, breaking changes, conflicts - **New plugin**: lands via PR following [`docs/CREATE_PLUGIN.md`](docs/CREATE_PLUGIN.md) - (description on every skill, ≥1 `evals/scenarios.json`, registered in + (description on every skill, registered in `marketplace.json` + `release-please-config.json`, a Habilité owner). Starts `experimental` → `release candidate` (one external success) → `stable` (Habilité review). diff --git a/UPGRADE.md b/UPGRADE.md index f6524cd5..02374068 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -20,7 +20,7 @@ This guide tells you exactly what disappears, what each old command becomes, and 1. **Delivery: marketplace, not clone.** In `v3` an external CLI copied the whole repo into each project and generated per-tool copies (Claude Code, Cursor, Copilot). In `v4` you point Claude Code at the marketplace and install plugins on demand. 2. **Split into 6 plugins.** Each plugin owns one slice of the SDLC and ships its own version. -3. **Commands became skills.** Every former `/command` is a skill with structured frontmatter, references, assets and evals. A skill can auto-trigger from your intent or be invoked by name. +3. **Commands became skills.** Every former `/command` is a skill with structured frontmatter, references, and assets. A skill can auto-trigger from your intent or be invoked by name. 4. **Many commands were merged into routers.** Related v3 commands (the three `assert_*`, the two `review_*`, `performance` + `security_refactor`, the three debug-family commands, the five `generate_*`) collapsed into a single skill that routes to the right sub-action. See the mapping in section 4. 5. **Agents, hooks, templates moved into their owning plugin** (for example `plugins/aidd-dev/agents/`). @@ -42,7 +42,7 @@ Each plugin ships: - `.claude-plugin/plugin.json` (manifest + version) - `skills/NN-action-name/SKILL.md` (one skill per former command or command family) - `CATALOG.md` (auto-generated index of the plugin's skills) -- its own assets, agents, references and evals +- its own assets, agents, and references --- diff --git a/aidd_docs/CONTRIBUTING.md b/aidd_docs/CONTRIBUTING.md index a356b89f..dab102ed 100644 --- a/aidd_docs/CONTRIBUTING.md +++ b/aidd_docs/CONTRIBUTING.md @@ -8,7 +8,7 @@ Use the generator skills to scaffold new content that follows the framework stru | Skill | Creates | | ---------------------------------- | -------------------- | -| `aidd-context:03-context-generate` | New skill, agent, or rule (router-based, with actions and evals) | +| `aidd-context:03-context-generate` | New skill, agent, or rule (router-based, with actions) | | `aidd-context:05-learn` | New memory or rule capturing a learning | Generator skills consume the templates inside their `assets/` folder and write the output to the correct location for your AI tool (Claude Code, Cursor, Copilot, Codex, OpenCode). @@ -19,7 +19,7 @@ All templates live alongside the skill that owns them, under `plugins//s | Where | What it scaffolds | | -------------------------------------------------- | -------------------------------------------------------- | -| `aidd-context:03-context-generate/assets/skills/` | `SKILL.md`, action, evals templates | +| `aidd-context:03-context-generate/assets/skills/` | `SKILL.md`, action templates | | `aidd-context:03-context-generate/assets/agents/` | Agent file template | | `aidd-context:03-context-generate/assets/rules/` | Rule file template | | `aidd-pm:03-prd/assets/` | PRD body template | @@ -38,7 +38,6 @@ When tools differ in syntax (frontmatter, slash command name, references), follo - Open a pull request for any new skill, agent, rule, or template. Visible changes that affect how the AI behaves on the project deserve team review. - Keep skills router-pure: SKILL.md holds no business logic; everything lives inside actions. -- Add evals (`evals/scenarios.json`) for every auto-trigger skill so router behavior stays correct over time. - Stay within 5 to 10 percent deviation from a template structure. Beyond that, update the template first, then derive the new content from it. ## Conventions diff --git a/aidd_docs/GUIDELINES.md b/aidd_docs/GUIDELINES.md index 96b306ef..489190f5 100644 --- a/aidd_docs/GUIDELINES.md +++ b/aidd_docs/GUIDELINES.md @@ -124,5 +124,4 @@ For each increment: - Anthropic, "Prompt engineering overview": - Anthropic, "Claude Code memory": - OpenAI, "Prompt engineering best practices": -- OpenAI Cookbook, "Eval-driven development": - GitHub Docs, "Repository custom instructions for Copilot": diff --git a/aidd_docs/README.md b/aidd_docs/README.md index 372e1f1d..f1fa3d4c 100644 --- a/aidd_docs/README.md +++ b/aidd_docs/README.md @@ -138,7 +138,6 @@ Inside the synchronous path, `aidd-dev:00-sdlc` is the Dev SDLC orchestrator tha - Skills must have an `## Available actions` table, `## Default flow`, `## Transversal rules`. - Actions must contain only `## Inputs`, `## Outputs`, `## Process`, `## Test`. - Tests must be observable: command, artifact check, or side effect. -- Evals (`evals/scenarios.json`) ship for every auto-trigger skill. --- diff --git a/aidd_docs/memory/architecture.md b/aidd_docs/memory/architecture.md index f6e1162b..33376837 100644 --- a/aidd_docs/memory/architecture.md +++ b/aidd_docs/memory/architecture.md @@ -44,7 +44,6 @@ flowchart TD Skills --> SkillMd["SKILL.md"] Skills --> Actions["actions/NN-action.md"] Skills --> Assets["assets/"] - Skills --> Evals["evals/scenarios.json"] ``` ## Services Communication diff --git a/aidd_docs/memory/project-brief.md b/aidd_docs/memory/project-brief.md index 51e3e484..59ff3097 100644 --- a/aidd_docs/memory/project-brief.md +++ b/aidd_docs/memory/project-brief.md @@ -28,7 +28,6 @@ AI-assisted software development tooling. The framework ships workflows, not cod | Memory | Structured context file loaded at conversation start | context file | | SDLC | Software Development Life Cycle — the end-to-end pipeline from idea to deployed PR | | | Marketplace | Central registry listing available plugins with version metadata | | -| Eval | Scenario JSON file verifying a skill's routing logic | test | ## Features & Use-cases diff --git a/aidd_docs/memory/testing.md b/aidd_docs/memory/testing.md index 8a0ccf99..648c2da2 100644 --- a/aidd_docs/memory/testing.md +++ b/aidd_docs/memory/testing.md @@ -2,20 +2,19 @@ ## Tools and Frameworks -- **Playwright MCP**: browser automation available via `.playwright-mcp/` config — used for manual or AI-driven UI testing on downstream projects -- **Evals** (`evals/scenarios.json`): per-skill JSON scenarios that verify routing logic (which action a skill dispatches to for a given input) +- **Playwright MCP**: browser automation available via `.playwright-mcp/` config, used for manual or AI-driven UI testing on downstream projects ## Testing Strategy - No unit test runner configured at framework level -- Skills are tested via eval scenarios; router behavior must be covered by `evals/scenarios.json` for every auto-trigger skill +- Skills are validated by running each action's `## Test` end-to-end against a real environment - Framework correctness validated by running actual skills against a real project (integration) ## Test Execution Process -- Evals: evaluated by the AI tool's eval runner or manually reviewed +- Each action declares a `## Test` (a command to run, an artifact check, or an observable side-effect) that must pass before the next action runs - `scripts/build-dist-verification.md` documents how to verify the build output ## Mocking and Stubbing -Not applicable — the framework has no runtime; all logic is markdown interpreted by an LLM. +Not applicable: the framework has no runtime; all logic is markdown interpreted by an LLM. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index b64644a3..bbdeb4bd 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -66,8 +66,7 @@ plugins// │ ├── README.md # human-facing skill landing │ ├── actions/ # atomic actions invoked by the router │ ├── assets/ # templates and static files -│ ├── references/ # extended docs the skill links into -│ └── evals/ # scenario fixtures +│ └── references/ # extended docs the skill links into ├── agents/ # named AI agents (optional) ├── commands/ # slash commands (optional) ├── hooks/hooks.json # lifecycle hooks (optional) diff --git a/docs/CREATE_PLUGIN.md b/docs/CREATE_PLUGIN.md index a24af98a..6b4a5b59 100644 --- a/docs/CREATE_PLUGIN.md +++ b/docs/CREATE_PLUGIN.md @@ -156,7 +156,6 @@ When you change SKILL.md or actions, run `/reload-plugins` in the same Claude Co ## Step 5 - document, test, ship - Run `pnpm exec lefthook run pre-commit` to confirm JSON validity, YAML validity, and SKILL.md frontmatter checks pass. The pre-commit hook also regenerates your `plugins/aidd-example/CATALOG.md` automatically. -- Add at least one scenario under `skills/01-hello/evals/scenarios.json` if you expect future evaluation runs. - Open a PR using the project template and pick the right commit scope (`feat(aidd-example): ...`). Listing the scope in `commitlint.config.cjs` is encouraged (an unlisted scope warns but does not block). ## Step 6 - release diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index dc359a25..3a155e64 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -16,7 +16,7 @@ Per-project context files under `aidd_docs/memory/` (architecture, conventions, ## Skill -A self-contained workflow under `plugins//skills//`. Triggered by a user phrase, a slash command, or an explicit `Use skill ` invocation. A skill owns a `SKILL.md` router, one or more atomic actions, optional `assets/` and `evals/`. The `SKILL.md` `name:` is the folder slug (`00-onboard`); the invocation id is `:`, for example `aidd-context:00-onboard`. +A self-contained workflow under `plugins//skills//`. Triggered by a user phrase, a slash command, or an explicit `Use skill ` invocation. A skill owns a `SKILL.md` router, one or more atomic actions, and optional `assets/` and `references/`. The `SKILL.md` `name:` is the folder slug (`00-onboard`); the invocation id is `:`, for example `aidd-context:00-onboard`. ## Router-based skill diff --git a/lefthook.yml b/lefthook.yml index b2dcdf1e..1277fc94 100644 --- a/lefthook.yml +++ b/lefthook.yml @@ -62,26 +62,6 @@ pre-commit: exit 1 fi done - evals-validity: - glob: "plugins/*/skills/*/evals/scenarios.json" - run: | - for f in {files}; do - [ -f "$f" ] || continue - if ! jq empty "$f" >/dev/null 2>&1; then - echo "❌ invalid JSON: $f" - exit 1 - fi - type=$(jq -r 'type' "$f") - if [ "$type" != "array" ]; then - echo "❌ $f: root must be an array, got $type" - exit 1 - fi - invalid=$(jq '[.[] | select((.prompt == null) or ((has("expect_action") | not) and (has("expect_skip") | not) and (has("expect") | not)))] | length' "$f") - if [ "$invalid" -gt 0 ]; then - echo "❌ $f: $invalid scenario(s) missing required keys (prompt + at least one of expect_action/expect_skip/expect)" - exit 1 - fi - done summarize-plugin-catalogs: run: | [ -d plugins ] || exit 0 diff --git a/plugins/aidd-context/CATALOG.md b/plugins/aidd-context/CATALOG.md index ca89f5a9..8c6c9223 100644 --- a/plugins/aidd-context/CATALOG.md +++ b/plugins/aidd-context/CATALOG.md @@ -8,7 +8,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai - [`.claude-plugin`](#claude-plugin) - [`hooks`](#hooks) - - [`hooks/routing`](#hooksrouting) - [`skills`](#skills) - [`skills/00-onboard`](#skills00-onboard) - [`skills/01-bootstrap`](#skills01-bootstrap) @@ -33,21 +32,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | [hooks.json](hooks/hooks.json) | | [update_memory.js](hooks/update_memory.js) | -#### `hooks/routing` - -| Group | File | -|-------|------| -| `bin` | [build-index.js](hooks/routing/bin/build-index.js) | -| `bin` | [build-on-session.js](hooks/routing/bin/build-on-session.js) | -| `bin` | [eval.js](hooks/routing/bin/eval.js) | -| `bin` | [measure-follow-rate.js](hooks/routing/bin/measure-follow-rate.js) | -| `bin` | [test-prompts.js](hooks/routing/bin/test-prompts.js) | -| `lib` | [bm25.js](hooks/routing/lib/bm25.js) | -| `lib` | [paths.js](hooks/routing/lib/paths.js) | -| `lib` | [tokenize.js](hooks/routing/lib/tokenize.js) | -| `-` | [routing-hint.js](hooks/routing/routing-hint.js) | -| `tests` | [held-out-prompts.json](hooks/routing/tests/held-out-prompts.json) | - ### `skills` #### `skills/00-onboard` @@ -58,7 +42,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-recommend-next.md](skills/00-onboard/actions/02-recommend-next.md) | - | | `actions` | [03-execute-or-handoff.md](skills/00-onboard/actions/03-execute-or-handoff.md) | - | | `assets` | [state-matrix.md](skills/00-onboard/assets/state-matrix.md) | - | -| `evals` | [scenarios.json](skills/00-onboard/evals/scenarios.json) | - | | `-` | [README.md](skills/00-onboard/README.md) | - | | `-` | [SKILL.md](skills/00-onboard/SKILL.md) | `Detect the current project's state and open a hub of project actions - understand the project, set up or refresh the memory bank, or continue the AIDD development journey. Silently inspects the project, the AIDD setup, and which AIDD plugins are installed, then adapts the menu to that context. Use when the user says "where do I start", "onboard me", "onboard me to this project", "what should I run next", "what should I work on next", "what's the state of this project", "guide me through aidd", "guide me through aidd-context", or invokes `aidd-context:00-onboard`. Do NOT use to enumerate every installed surface from raw user intent (the discovery skill in this plugin handles that).` | @@ -73,7 +56,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [05-write-install-md.md](skills/01-bootstrap/actions/05-write-install-md.md) | - | | `assets` | [checklist.md](skills/01-bootstrap/assets/checklist.md) | - | | `assets` | [install-template.md](skills/01-bootstrap/assets/install-template.md) | - | -| `evals` | [scenarios.json](skills/01-bootstrap/evals/scenarios.json) | - | | `-` | [README.md](skills/01-bootstrap/README.md) | - | | `references` | [stack-heuristics.md](skills/01-bootstrap/references/stack-heuristics.md) | - | | `-` | [SKILL.md](skills/01-bootstrap/SKILL.md) | `Imagine and validate the technical architecture of a new SaaS through interactive Q&A, candidate-stack comparison, multi-agent audit, and an INSTALL.md output. Use when starting a new SaaS project, choosing a stack, designing the architecture pattern (monolith vs microservices vs serverless), or producing a project's INSTALL.md. Do NOT use for editing an existing project's stack, database schema design, or scaffolding actual files (this skill produces docs only, no code).` | @@ -92,7 +74,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `assets` | [golden-principles.md](skills/02-project-init/assets/golden-principles.md) | - | | `assets` | [GUIDELINES.md](skills/02-project-init/assets/GUIDELINES.md) | - | | `assets` | [README.md](skills/02-project-init/assets/README.md) | - | -| `evals` | [scenarios.json](skills/02-project-init/evals/scenarios.json) | - | | `-` | [README.md](skills/02-project-init/README.md) | - | | `references` | [mapping-ai-context-file.md](skills/02-project-init/references/mapping-ai-context-file.md) | - | | `-` | [SKILL.md](skills/02-project-init/SKILL.md) | `Initialize or refresh the project memory bank and ensure AI context files contain the project memory block. Use when running `aidd init` for the first time, bootstrapping a new project, or re-running the init flow on an existing project. Do NOT use for updating individual memory files after they exist - use `aidd-context:05-learn` instead; do NOT use for editing a single rule - edit the file directly.` | @@ -101,7 +82,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| -| `evals` | [scenarios.json](skills/03-context-generate/evals/scenarios.json) | - | | `-` | [README.md](skills/03-context-generate/README.md) | - | | `references` | [ai-mapping.md](skills/03-context-generate/references/ai-mapping.md) | - | | `references` | [command.md](skills/03-context-generate/references/command.md) | - | @@ -117,7 +97,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| | `actions` | [01-mermaid.md](skills/04-mermaid/actions/01-mermaid.md) | - | -| `evals` | [scenarios.json](skills/04-mermaid/evals/scenarios.json) | - | | `-` | [README.md](skills/04-mermaid/README.md) | - | | `references` | [mermaid-conventions.md](skills/04-mermaid/references/mermaid-conventions.md) | - | | `-` | [SKILL.md](skills/04-mermaid/SKILL.md) | `Generate high-quality Mermaid diagrams from markdown content using a structured plan-validate workflow.` | @@ -131,7 +110,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [03-sync.md](skills/05-learn/actions/03-sync.md) | - | - | | `assets` | [adr-template.md](skills/05-learn/assets/adr-template.md) | `Architecture Decision Record template` | - | | `assets` | [decision-template.md](skills/05-learn/assets/decision-template.md) | `Individual decision record template` | `` | -| `evals` | [scenarios.json](skills/05-learn/evals/scenarios.json) | - | - | | `-` | [README.md](skills/05-learn/README.md) | - | - | | `-` | [SKILL.md](skills/05-learn/SKILL.md) | `Capture and store project-level learnings, conventions, and decisions surfaced during work into memory, decisions, or rules. Use proactively when the user states a durable project rule or convention ("for next", "always do X", "from now on", "going forward", "rule:", "convention:"), records a technical decision and its rationale, deprecates something, or notes an insight that should outlive the current task. Do NOT use for personal or AI-preference reminders (those belong to user memory), routine code edits, minor fixes, or anything already captured.` | - | @@ -147,7 +125,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [06-find-rule.md](skills/06-discovery/actions/06-find-rule.md) | - | | `actions` | [07-find-hook.md](skills/06-discovery/actions/07-find-hook.md) | - | | `actions` | [08-find-memory.md](skills/06-discovery/actions/08-find-memory.md) | - | -| `evals` | [scenarios.json](skills/06-discovery/evals/scenarios.json) | - | | `-` | [README.md](skills/06-discovery/README.md) | - | | `references` | [ai-mapping.md](skills/06-discovery/references/ai-mapping.md) | - | | `-` | [SKILL.md](skills/06-discovery/SKILL.md) | `Enumerate installed surfaces of the AI tool (skills, agents, commands, plugins, MCP servers, rules, hooks, memory files) and recommend the best match for the user's stated intent. Use proactively whenever the user asks the model to list, show, enumerate, find, or pick among any of these surfaces - including imperative phrasings ("list hooks", "show me the rules", "enumerate skills", "find a memory file", "which agent reviews code"), question phrasings ("what's available?", "what hooks do we have?", "which rule applies here?", "what memory files do we have?"), and indirect phrasings ("what can I use for X?", "do we have something that does Y?"). Always pick this skill over scanning the filesystem with grep, find, ls, or reading action files directly when the user is enumerating a surface. Do NOT use for picking a specific item inside one plugin (the plugin's own onboard handles that), creating a new surface, or executing a recommended item (this skill only points; the user invokes).` | diff --git a/plugins/aidd-context/hooks/hooks.json b/plugins/aidd-context/hooks/hooks.json index 78c434ab..76b2fb90 100644 --- a/plugins/aidd-context/hooks/hooks.json +++ b/plugins/aidd-context/hooks/hooks.json @@ -6,22 +6,6 @@ { "type": "command", "command": "node ${CLAUDE_PLUGIN_ROOT}/hooks/update_memory.js" - }, - { - "type": "command", - "command": "node ${CLAUDE_PLUGIN_ROOT}/hooks/routing/bin/build-on-session.js", - "timeout": 30 - } - ] - } - ], - "UserPromptSubmit": [ - { - "hooks": [ - { - "type": "command", - "command": "node ${CLAUDE_PLUGIN_ROOT}/hooks/routing/routing-hint.js", - "timeout": 5 } ] } diff --git a/plugins/aidd-context/hooks/routing/bin/build-index.js b/plugins/aidd-context/hooks/routing/bin/build-index.js deleted file mode 100644 index 394368f8..00000000 --- a/plugins/aidd-context/hooks/routing/bin/build-index.js +++ /dev/null @@ -1,274 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const fs = require('fs'); -const path = require('path'); -const crypto = require('crypto'); -const { tokenize } = require('../lib/tokenize'); -const { buildIndex, scoreDoc } = require('../lib/bm25'); -const { INSTALLED_PLUGINS, KNOWN_MARKETPLACES, dataDir } = require('../lib/paths'); - -const MIN_POS = 8; -const MIN_NEG = 2; - -function parseArgs() { - const a = process.argv.slice(2); - const opts = { plugins: null, out: null, dryRun: false }; - for (let i = 0; i < a.length; i++) { - if (a[i] === '--plugins') opts.plugins = a[++i].split(','); - else if (a[i] === '--out') opts.out = a[++i]; - else if (a[i] === '--dry-run') opts.dryRun = true; - } - return opts; -} - -function tryReadJson(p) { - try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return null; } -} - -function resolvePluginDir(pluginName, marketplaceName, instances, markets) { - const market = markets[marketplaceName]; - if (market?.source?.source === 'directory' && market.installLocation) { - const marketDir = market.installLocation; - const mp = tryReadJson(path.join(marketDir, '.claude-plugin', 'marketplace.json')); - const entry = mp?.plugins?.find(p => p.name === pluginName); - if (entry?.source) { - const dir = path.resolve(marketDir, entry.source); - if (fs.existsSync(dir)) return { dir, mode: 'directory-source' }; - } - } - if (instances?.[0]?.installPath && fs.existsSync(instances[0].installPath)) { - return { dir: instances[0].installPath, mode: 'cache' }; - } - return null; -} - -function findScenarios(pluginFilter) { - if (!fs.existsSync(INSTALLED_PLUGINS)) return []; - const installed = tryReadJson(INSTALLED_PLUGINS); - const markets = tryReadJson(KNOWN_MARKETPLACES) || {}; - if (!installed) return []; - - const out = []; - for (const [pluginKey, instances] of Object.entries(installed.plugins || {})) { - const [pluginName, marketplaceName] = pluginKey.split('@'); - if (pluginFilter && !pluginFilter.includes(pluginName)) continue; - - const resolved = resolvePluginDir(pluginName, marketplaceName, instances, markets); - if (!resolved) continue; - - const skillsDir = path.join(resolved.dir, 'skills'); - if (!fs.existsSync(skillsDir)) continue; - let skills; - try { skills = fs.readdirSync(skillsDir); } catch { continue; } - for (const skill of skills) { - const evalsFile = path.join(skillsDir, skill, 'evals', 'scenarios.json'); - if (!fs.existsSync(evalsFile)) continue; - const data = tryReadJson(evalsFile); - if (!Array.isArray(data)) continue; - const skillId = `${pluginName}:${skill}`; - const mtime = fs.statSync(evalsFile).mtimeMs; - for (const entry of data) { - if (!entry || typeof entry.prompt !== 'string') continue; - out.push({ - skill: skillId, - prompt: entry.prompt, - expect_action: entry.expect_action, - polarity: entry.expect_action == null ? 'neg' : 'pos', - _source: evalsFile, - _mtime: mtime, - _mode: resolved.mode, - }); - } - } - } - return out; -} - -function tokenizeAll(scenarios) { - for (const s of scenarios) { - const tok = tokenize(s.prompt); - s.tokens = tok.tokens; - s.negated = tok.negated; - } - return scenarios; -} - -function scorePositiveAgainstSkill(p, skillPositives, idf, avgdl, looSelf = false) { - let best = 0; - for (const doc of skillPositives) { - if (looSelf && doc === p) continue; - const s = scoreDoc(p.tokens, doc, idf, avgdl); - if (s > best) best = s; - } - return best; -} - -function f1AtThreshold(posScores, negScores, tau) { - const TP = posScores.filter(s => s >= tau).length; - const FN = posScores.length - TP; - const FP = negScores.filter(s => s >= tau).length; - const precision = TP + FP === 0 ? 0 : TP / (TP + FP); - const recall = TP + FN === 0 ? 0 : TP / (TP + FN); - const f1 = precision + recall === 0 ? 0 : 2 * precision * recall / (precision + recall); - const beta2 = 4; - const fbeta2 = beta2 * precision + recall === 0 - ? 0 - : (1 + beta2) * precision * recall / (beta2 * precision + recall); - return { TP, FP, FN, precision, recall, f1, fbeta2 }; -} - -function calibrateThresholds(scenarios, index) { - const skills = [...new Set(scenarios.map(s => s.skill))]; - const thresholds = {}; - for (const skillId of skills) { - const skillScenarios = scenarios.filter(s => s.skill === skillId); - const positives = skillScenarios.filter(s => s.polarity === 'pos'); - const negatives = skillScenarios.filter(s => s.polarity === 'neg'); - if (positives.length < MIN_POS || negatives.length < MIN_NEG) { - thresholds[skillId] = { - tau: null, f1: null, - n_pos: positives.length, n_neg: negatives.length, - status: 'excluded', - }; - continue; - } - const posScores = positives.map(p => - scorePositiveAgainstSkill(p, positives, index.idf, index.avgdl, true)); - const negScores = negatives.map(n => - scorePositiveAgainstSkill(n, positives, index.idf, index.avgdl, false)); - - const candidates = [...new Set([...posScores, ...negScores])] - .filter(c => c > 0).sort((a, b) => a - b); - if (candidates.length === 0) { - thresholds[skillId] = { - tau: null, f1: 0, recall: 0, precision: 0, - n_pos: positives.length, n_neg: negatives.length, status: 'conflict', - }; - continue; - } - let bestFbeta = -1, bestTau = 0, bestMetrics = null; - for (let i = 0; i < candidates.length; i++) { - const tau = i === 0 ? candidates[0] - 0.001 : (candidates[i - 1] + candidates[i]) / 2; - if (tau <= 0) continue; - const m = f1AtThreshold(posScores, negScores, tau); - if (m.fbeta2 > bestFbeta) { - bestFbeta = m.fbeta2; bestTau = tau; bestMetrics = m; - } - } - thresholds[skillId] = { - tau: bestTau, - f1: bestMetrics.f1, - fbeta2: bestMetrics.fbeta2, - precision: bestMetrics.precision, - recall: bestMetrics.recall, - TP: bestMetrics.TP, FP: bestMetrics.FP, FN: bestMetrics.FN, - n_pos: positives.length, n_neg: negatives.length, - status: bestMetrics.f1 >= 0.60 ? 'ok' : 'conflict', - }; - } - return thresholds; -} - -function buildHash(scenarios, index) { - const h = crypto.createHash('sha256'); - const sorted = [...scenarios].sort((a, b) => - (a.skill + a.prompt).localeCompare(b.skill + b.prompt)); - for (const s of sorted) h.update(`${s.skill}|${s.prompt}|${s.expect_action}\n`); - h.update(JSON.stringify({ K1: index.K1, B: index.B })); - return h.digest('hex'); -} - -function cacheKey(scenarios) { - const h = crypto.createHash('sha256'); - if (fs.existsSync(INSTALLED_PLUGINS)) h.update(fs.readFileSync(INSTALLED_PLUGINS)); - const sources = [...new Set(scenarios.map(s => s._source).filter(Boolean))].sort(); - for (const src of sources) { - const stat = fs.statSync(src); - h.update(`${src}|${stat.mtimeMs}\n`); - } - return h.digest('hex'); -} - -function main() { - const opts = parseArgs(); - const outDir = opts.out ? path.resolve(opts.out) : dataDir(); - - const scenarios = tokenizeAll(findScenarios(opts.plugins)); - if (scenarios.length === 0) { - console.error(`[build-index] no scenarios found (out=${outDir})`); - process.exit(1); - } - - const ckey = cacheKey(scenarios); - const manifestPath = path.join(outDir, 'manifest.json'); - const existing = tryReadJson(manifestPath); - if (existing && existing.cache_key === ckey && !opts.dryRun) { - try { - const tmp = manifestPath + '.tmp.' + process.pid; - fs.writeFileSync(tmp, JSON.stringify( - { ...existing, built_at: new Date().toISOString() }, null, 2)); - fs.renameSync(tmp, manifestPath); - } catch { /* ignore */ } - console.log(`[build-index] cache hit (key=${ckey.slice(0, 12)}), skip rebuild`); - return; - } - - const index = buildIndex(scenarios); - const thresholds = calibrateThresholds(scenarios, index); - const hash = buildHash(scenarios, index); - - if (opts.dryRun) { - console.log(`[build-index] dry-run scenarios=${scenarios.length} skills=${Object.keys(thresholds).length} cache_key=${ckey.slice(0, 12)}`); - return; - } - - fs.mkdirSync(outDir, { recursive: true }); - - const writeAtomic = (file, content) => { - const tmp = file + '.tmp.' + process.pid; - fs.writeFileSync(tmp, content); - fs.renameSync(tmp, file); - }; - - writeAtomic(path.join(outDir, 'index.json'), JSON.stringify({ - version: 1, - build_hash: hash, - params: { K1: index.K1, B: index.B }, - avgdl: index.avgdl, - idf: index.idf, - scenarios: scenarios.map(s => ({ - skill: s.skill, prompt: s.prompt, - tokens: s.tokens, polarity: s.polarity, - })), - }, null, 2)); - - writeAtomic(path.join(outDir, 'thresholds.json'), - JSON.stringify(thresholds, null, 2)); - - writeAtomic(path.join(outDir, 'manifest.json'), JSON.stringify({ - version: 1, - build_hash: hash, - cache_key: ckey, - params: { K1: index.K1, B: index.B }, - plugins_filter: opts.plugins, - scenarios_count: scenarios.length, - positives: scenarios.filter(s => s.polarity === 'pos').length, - negatives: scenarios.filter(s => s.polarity === 'neg').length, - skills: Object.keys(thresholds).length, - excluded_skills: Object.entries(thresholds) - .filter(([, t]) => t.status === 'excluded').map(([k]) => k), - conflict_skills: Object.entries(thresholds) - .filter(([, t]) => t.status === 'conflict').map(([k]) => k), - built_at: new Date().toISOString(), - }, null, 2)); - - console.log(`[build-index] built scenarios=${scenarios.length} skills=${Object.keys(thresholds).length} hash=${hash.slice(0, 12)}`); - console.log(`[build-index] out=${outDir}`); - const conflicts = Object.entries(thresholds).filter(([, t]) => t.status === 'conflict'); - if (conflicts.length) console.log(`[build-index] conflicts: ${conflicts.map(([k]) => k).join(', ')}`); - const excluded = Object.entries(thresholds).filter(([, t]) => t.status === 'excluded'); - if (excluded.length) console.log(`[build-index] excluded: ${excluded.length} skills`); -} - -main(); diff --git a/plugins/aidd-context/hooks/routing/bin/build-on-session.js b/plugins/aidd-context/hooks/routing/bin/build-on-session.js deleted file mode 100644 index 40b3d0b3..00000000 --- a/plugins/aidd-context/hooks/routing/bin/build-on-session.js +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const { spawnSync } = require('child_process'); -const path = require('path'); -const fs = require('fs'); -const os = require('os'); - -const BUILD_SCRIPT = path.join(__dirname, 'build-index.js'); -const LOG_DIR = process.env.CLAUDE_PLUGIN_DATA - ? path.join(process.env.CLAUDE_PLUGIN_DATA, 'routing') - : path.join(os.homedir(), '.claude', 'plugins', 'data', 'aidd-context', 'routing'); - -function passthrough() { process.exit(0); } - -function main() { - try { fs.mkdirSync(LOG_DIR, { recursive: true }); } catch { /* ignore */ } - const logPath = path.join(LOG_DIR, 'last-build.log'); - - const r = spawnSync(process.execPath, [BUILD_SCRIPT], { - encoding: 'utf8', - timeout: 25000, - }); - - const log = [ - `[${new Date().toISOString()}] build-on-session`, - `exit ${r.status}`, - '--- stdout ---', - r.stdout || '', - '--- stderr ---', - r.stderr || '', - ].join('\n'); - - try { fs.writeFileSync(logPath, log); } catch { /* ignore */ } - passthrough(); -} - -try { main(); } catch { passthrough(); } diff --git a/plugins/aidd-context/hooks/routing/bin/eval.js b/plugins/aidd-context/hooks/routing/bin/eval.js deleted file mode 100644 index b0793e69..00000000 --- a/plugins/aidd-context/hooks/routing/bin/eval.js +++ /dev/null @@ -1,183 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const fs = require('fs'); -const path = require('path'); -const { scoreDoc } = require('../lib/bm25'); -const { dataDir } = require('../lib/paths'); - -const DATA_DIR = dataDir(); - -const F1_GLOBAL_MIN = 0.70; -const F1_PER_SKILL_MIN = 0.55; -const COLLISION_MAX = 0.60; - -function loadJson(p) { return JSON.parse(fs.readFileSync(p, 'utf8')); } - -function predictLOO(scenario, allScenarios, idf, avgdl) { - const others = allScenarios.filter(s => - s.polarity === 'pos' && !(s.skill === scenario.skill && s.prompt === scenario.prompt)); - const bySkill = new Map(); - for (const doc of others) { - const score = scoreDoc(scenario.tokens, doc, idf, avgdl); - const cur = bySkill.get(doc.skill) || 0; - if (score > cur) bySkill.set(doc.skill, score); - } - return bySkill; -} - -function evalAll(index, thresholds) { - const { idf, avgdl, scenarios } = index; - const perSkill = {}; - for (const skillId of Object.keys(thresholds)) { - perSkill[skillId] = { TP: 0, FP: 0, FN: 0, TN: 0, n_pos: 0, n_neg: 0 }; - } - - for (const s of scenarios) { - const scores = predictLOO(s, scenarios, idf, avgdl); - const matched = new Set(); - for (const [skill, score] of scores) { - const t = thresholds[skill]; - if (!t || t.status === 'excluded' || t.tau == null) continue; - if (score >= t.tau) matched.add(skill); - } - - if (s.polarity === 'pos') { - const expected = s.skill; - perSkill[expected].n_pos++; - if (matched.has(expected)) perSkill[expected].TP++; - else perSkill[expected].FN++; - for (const m of matched) { - if (m !== expected) perSkill[m].FP++; - } - } else { - const owner = s.skill; - perSkill[owner].n_neg++; - if (matched.has(owner)) perSkill[owner].FP++; - else perSkill[owner].TN++; - } - } - - const results = {}; - let globalTP = 0, globalFP = 0, globalFN = 0; - for (const [skill, m] of Object.entries(perSkill)) { - const t = thresholds[skill]; - if (t.status === 'excluded') { - results[skill] = { status: 'excluded', n_pos: m.n_pos, n_neg: m.n_neg }; - continue; - } - const precision = m.TP + m.FP === 0 ? 0 : m.TP / (m.TP + m.FP); - const recall = m.TP + m.FN === 0 ? 0 : m.TP / (m.TP + m.FN); - const f1 = precision + recall === 0 ? 0 : 2 * precision * recall / (precision + recall); - results[skill] = { - precision: round(precision), recall: round(recall), f1: round(f1), - TP: m.TP, FP: m.FP, FN: m.FN, TN: m.TN, - tau: round(t.tau), - n_pos: m.n_pos, n_neg: m.n_neg, - }; - globalTP += m.TP; globalFP += m.FP; globalFN += m.FN; - } - const gp = globalTP + globalFP === 0 ? 0 : globalTP / (globalTP + globalFP); - const gr = globalTP + globalFN === 0 ? 0 : globalTP / (globalTP + globalFN); - const gf1 = gp + gr === 0 ? 0 : 2 * gp * gr / (gp + gr); - - return { perSkill: results, global: { precision: round(gp), recall: round(gr), f1: round(gf1) } }; -} - -function collisionMatrix(index, thresholds) { - const { idf, avgdl, scenarios } = index; - const skills = Object.keys(thresholds).filter(s => thresholds[s].status !== 'excluded'); - const matrix = {}; - for (const expected of skills) { - matrix[expected] = {}; - const ownPos = scenarios.filter(s => s.skill === expected && s.polarity === 'pos'); - for (const target of skills) { - const targetPos = scenarios.filter(s => s.skill === target && s.polarity === 'pos'); - let sum = 0, count = 0; - for (const q of ownPos) { - let best = 0; - for (const doc of targetPos) { - if (q === doc) continue; - const s = scoreDoc(q.tokens, doc, idf, avgdl); - if (s > best) best = s; - } - const tau = thresholds[target].tau; - sum += best >= tau ? 1 : 0; - count++; - } - matrix[expected][target] = count ? round(sum / count) : 0; - } - } - return matrix; -} - -function round(x, d = 3) { return Math.round(x * 10 ** d) / 10 ** d; } - -function printTable(perSkill, global) { - console.log('\nPer-skill (LOO):'); - console.log('skill P R F1 TP FP FN TN τ'); - for (const [skill, r] of Object.entries(perSkill)) { - if (r.status === 'excluded') { - console.log(`${skill.padEnd(38)} excluded (n_pos=${r.n_pos}, n_neg=${r.n_neg})`); - continue; - } - console.log(`${skill.padEnd(38)} ${pad(r.precision)} ${pad(r.recall)} ${pad(r.f1)} ${pad2(r.TP)} ${pad2(r.FP)} ${pad2(r.FN)} ${pad2(r.TN)} ${r.tau}`); - } - console.log(`\nGlobal: P=${global.precision} R=${global.recall} F1=${global.f1}`); -} - -function pad(x) { return x.toFixed(3).padEnd(6); } -function pad2(x) { return String(x).padStart(2); } - -function printCollision(matrix) { - const skills = Object.keys(matrix); - if (skills.length === 0) return; - console.log('\nCollision matrix (rows=query skill, cols=target skill — fraction matched above τ):'); - const header = ' ' + skills.map(s => s.split(':').pop().padStart(8)).join(' '); - console.log(header); - for (const r of skills) { - const row = r.padEnd(38) + skills.map(c => round(matrix[r][c]).toFixed(2).padStart(8)).join(' '); - console.log(row); - } -} - -function checkGate(results, matrix) { - const failures = []; - if (results.global.f1 < F1_GLOBAL_MIN) { - failures.push(`global F1 ${results.global.f1} < ${F1_GLOBAL_MIN}`); - } - for (const [skill, r] of Object.entries(results.perSkill)) { - if (r.status === 'excluded') continue; - if (r.f1 < F1_PER_SKILL_MIN) { - failures.push(`${skill} F1 ${r.f1} < ${F1_PER_SKILL_MIN}`); - } - } - for (const [row, cols] of Object.entries(matrix)) { - for (const [col, v] of Object.entries(cols)) { - if (row !== col && v > COLLISION_MAX) { - failures.push(`collision ${row} → ${col}: ${v} > ${COLLISION_MAX}`); - } - } - } - return failures; -} - -function main() { - const index = loadJson(path.join(DATA_DIR, 'index.json')); - const thresholds = loadJson(path.join(DATA_DIR, 'thresholds.json')); - const results = evalAll(index, thresholds); - const matrix = collisionMatrix(index, thresholds); - printTable(results.perSkill, results.global); - printCollision(matrix); - const failures = checkGate(results, matrix); - fs.writeFileSync(path.join(DATA_DIR, 'metrics.json'), - JSON.stringify({ results, collision: matrix, gate: { passed: failures.length === 0, failures } }, null, 2)); - if (failures.length) { - console.log('\nGATE FAILED:'); - failures.forEach(f => console.log(` - ${f}`)); - process.exit(1); - } - console.log('\nGate PASSED ✓'); -} - -main(); diff --git a/plugins/aidd-context/hooks/routing/bin/measure-follow-rate.js b/plugins/aidd-context/hooks/routing/bin/measure-follow-rate.js deleted file mode 100644 index 6480a265..00000000 --- a/plugins/aidd-context/hooks/routing/bin/measure-follow-rate.js +++ /dev/null @@ -1,274 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const fs = require('fs'); -const os = require('os'); -const path = require('path'); - -const PROJECTS_DIR = path.join(os.homedir(), '.claude', 'projects'); - -function parseArgs() { - const a = process.argv.slice(2); - const opts = { since: null, projectFilter: null, verbose: false }; - for (let i = 0; i < a.length; i++) { - if (a[i] === '--since') opts.since = new Date(a[++i]).getTime(); - else if (a[i] === '--project') opts.projectFilter = a[++i]; - else if (a[i] === '-v' || a[i] === '--verbose') opts.verbose = true; - } - return opts; -} - -function listSessions(opts) { - if (!fs.existsSync(PROJECTS_DIR)) return []; - const sessions = []; - for (const projectDir of fs.readdirSync(PROJECTS_DIR)) { - if (opts.projectFilter && !projectDir.includes(opts.projectFilter)) continue; - const fullDir = path.join(PROJECTS_DIR, projectDir); - if (!fs.statSync(fullDir).isDirectory()) continue; - for (const file of fs.readdirSync(fullDir)) { - if (!file.endsWith('.jsonl')) continue; - const fullFile = path.join(fullDir, file); - const stat = fs.statSync(fullFile); - if (opts.since && stat.mtimeMs < opts.since) continue; - sessions.push({ project: projectDir, file: fullFile, mtime: stat.mtimeMs }); - } - } - return sessions.sort((a, b) => b.mtime - a.mtime); -} - -function readJsonl(file) { - const lines = fs.readFileSync(file, 'utf8').split(/\r?\n/); - const out = []; - for (const line of lines) { - if (!line.trim()) continue; - try { out.push(JSON.parse(line)); } catch { /* skip malformed */ } - } - return out; -} - -function extractText(message) { - if (!message) return ''; - const c = message.message?.content; - if (typeof c === 'string') return c; - if (Array.isArray(c)) { - return c.filter(p => p?.type === 'text' && typeof p.text === 'string') - .map(p => p.text).join('\n'); - } - return ''; -} - -function parseHintContent(content) { - if (!Array.isArray(content)) return null; - const text = content.find(s => typeof s === 'string' && /Routing hint/i.test(s)); - if (!text) return null; - const skills = []; - const re = /-\s+([\w\-:]+)\s+\((\d+)%\s+confidence/g; - let m; - while ((m = re.exec(text)) !== null) { - skills.push({ skill: m[1], confidence: parseInt(m[2], 10) }); - } - if (skills.length === 0) { - const reSimple = /-\s+([\w\-:]+)\s+\(score/g; - while ((m = reSimple.exec(text)) !== null) skills.push({ skill: m[1], confidence: null }); - } - return skills.length ? skills : null; -} - -function findLaunchSkill(content) { - if (typeof content === 'string') { - const m = content.match(/Launching skill:\s*([\w\-:]+)/); - return m ? m[1] : null; - } - if (Array.isArray(content)) { - for (const part of content) { - if (part?.type === 'text' && part.text) { - const m = part.text.match(/Launching skill:\s*([\w\-:]+)/); - if (m) return m[1]; - } - if (part?.type === 'tool_result' && typeof part.content === 'string') { - const m = part.content.match(/Launching skill:\s*([\w\-:]+)/); - if (m) return m[1]; - } - if (part?.type === 'tool_result' && Array.isArray(part.content)) { - for (const sub of part.content) { - if (typeof sub === 'string') { - const m = sub.match(/Launching skill:\s*([\w\-:]+)/); - if (m) return m[1]; - } - if (sub?.type === 'text' && sub.text) { - const m = sub.text.match(/Launching skill:\s*([\w\-:]+)/); - if (m) return m[1]; - } - } - } - } - } - return null; -} - -function isCommandWrapper(content) { - if (typeof content !== 'string') return false; - return /^<(local-command-|command-name)/.test(content); -} - -function analyzeSession(messages) { - const turns = []; - let pendingUserPrompt = null; - let pendingHint = null; - - const flushPending = () => { - if (pendingUserPrompt !== null) { - turns.push({ prompt: pendingUserPrompt, hint: pendingHint, skill_invoked: null }); - pendingUserPrompt = null; - pendingHint = null; - } - }; - - for (const m of messages) { - if (m.type === 'user' && m.message?.role === 'user') { - const content = m.message.content; - const launched = findLaunchSkill(content); - if (launched) { - flushPending(); - const last = turns[turns.length - 1]; - if (last && !last.skill_invoked) last.skill_invoked = launched; - continue; - } - if (typeof content === 'string' && content && !isCommandWrapper(content)) { - flushPending(); - pendingUserPrompt = content; - } - } else if (m.type === 'attachment' && m.attachment?.hookName === 'UserPromptSubmit') { - const skills = parseHintContent(m.attachment.content); - if (skills) pendingHint = skills; - } else if (m.type === 'assistant') { - flushPending(); - const lastTurn = turns[turns.length - 1]; - if (lastTurn) { - const launched = findLaunchSkill(m.message?.content); - if (launched && !lastTurn.skill_invoked) lastTurn.skill_invoked = launched; - } - } - } - flushPending(); - return turns; -} - -function classifyTurn(t) { - const hintTop1 = t.hint?.[0]?.skill || null; - const hintAll = t.hint ? t.hint.map(s => s.skill) : []; - if (!t.hint && !t.skill_invoked) return 'no-hint-no-skill'; - if (!t.hint && t.skill_invoked) return 'missed-by-router'; - if (t.hint && !t.skill_invoked) return 'hint-ignored'; - if (hintTop1 === t.skill_invoked) return 'follow-top1'; - if (hintAll.includes(t.skill_invoked)) return 'follow-in-topk'; - return 'wrong-route'; -} - -function round(x, d = 1) { return Math.round(x * 10 ** d) / 10 ** d; } - -function main() { - const opts = parseArgs(); - const sessions = listSessions(opts); - if (sessions.length === 0) { - console.error('No sessions found.'); - process.exit(1); - } - - const overall = { - 'follow-top1': 0, - 'follow-in-topk': 0, - 'wrong-route': 0, - 'hint-ignored': 0, - 'missed-by-router': 0, - 'no-hint-no-skill': 0, - }; - const bySkillInvoked = {}; - const examples = { 'wrong-route': [], 'hint-ignored': [], 'missed-by-router': [] }; - - let totalSessions = 0; - let totalTurns = 0; - - for (const s of sessions) { - const messages = readJsonl(s.file); - if (messages.length === 0) continue; - totalSessions++; - const turns = analyzeSession(messages); - for (const t of turns) { - totalTurns++; - const verdict = classifyTurn(t); - overall[verdict]++; - - const skill = t.skill_invoked || '(none)'; - if (!bySkillInvoked[skill]) { - bySkillInvoked[skill] = { 'follow-top1': 0, 'follow-in-topk': 0, 'wrong-route': 0, 'missed-by-router': 0, total: 0 }; - } - bySkillInvoked[skill][verdict] = (bySkillInvoked[skill][verdict] || 0) + 1; - bySkillInvoked[skill].total++; - - if (examples[verdict] && examples[verdict].length < 5) { - examples[verdict].push({ - prompt: (t.prompt || '').slice(0, 60), - hint_top1: t.hint?.[0]?.skill || null, - skill_invoked: t.skill_invoked, - session: path.basename(s.file, '.jsonl').slice(0, 8), - }); - } - } - } - - console.log(`\nSessions analyzed: ${totalSessions}`); - console.log(`Total turns: ${totalTurns}\n`); - - const hinted = overall['follow-top1'] + overall['follow-in-topk'] + overall['wrong-route'] + overall['hint-ignored']; - const followed = overall['follow-top1'] + overall['follow-in-topk']; - const followRate = hinted ? round(followed / hinted * 100) : 0; - const top1Rate = hinted ? round(overall['follow-top1'] / hinted * 100) : 0; - - console.log('=== Overall ==='); - console.log(`hint emitted : ${hinted}`); - console.log(` follow top-1 : ${overall['follow-top1']} (${hinted ? round(overall['follow-top1'] / hinted * 100) : 0}%)`); - console.log(` follow in top-K : ${overall['follow-in-topk']} (${hinted ? round(overall['follow-in-topk'] / hinted * 100) : 0}%)`); - console.log(` wrong route : ${overall['wrong-route']} (Claude invoked a skill NOT in hint)`); - console.log(` hint ignored : ${overall['hint-ignored']} (Claude invoked no skill despite hint)`); - console.log(`no hint emitted : ${overall['missed-by-router'] + overall['no-hint-no-skill']}`); - console.log(` missed by router : ${overall['missed-by-router']} (Claude invoked skill we did NOT hint)`); - console.log(` no-skill-needed : ${overall['no-hint-no-skill']}`); - - console.log(`\nFollow rate (top-K) : ${followRate}%`); - console.log(`Top-1 match rate : ${top1Rate}%`); - - console.log('\n=== By skill invoked ==='); - const skills = Object.entries(bySkillInvoked) - .filter(([k]) => k !== '(none)') - .sort((a, b) => b[1].total - a[1].total) - .slice(0, 20); - if (skills.length === 0) { - console.log('(no skill invocations found)'); - } else { - console.log('skill total top1 topk wrong missed'); - for (const [skill, s] of skills) { - console.log(`${skill.padEnd(34)} ${String(s.total).padStart(5)} ${String(s['follow-top1'] || 0).padStart(4)} ${String(s['follow-in-topk'] || 0).padStart(4)} ${String(s['wrong-route'] || 0).padStart(5)} ${String(s['missed-by-router'] || 0).padStart(6)}`); - } - } - - if (opts.verbose) { - console.log('\n=== Sample wrong-routes ==='); - for (const ex of examples['wrong-route']) { - console.log(` [${ex.session}] "${ex.prompt}"`); - console.log(` hint top-1: ${ex.hint_top1} → invoked: ${ex.skill_invoked}`); - } - console.log('\n=== Sample hint-ignored ==='); - for (const ex of examples['hint-ignored']) { - console.log(` [${ex.session}] "${ex.prompt}"`); - console.log(` hint top-1: ${ex.hint_top1} → invoked: nothing`); - } - console.log('\n=== Sample missed-by-router ==='); - for (const ex of examples['missed-by-router']) { - console.log(` [${ex.session}] "${ex.prompt}"`); - console.log(` hint: none → invoked: ${ex.skill_invoked}`); - } - } -} - -main(); diff --git a/plugins/aidd-context/hooks/routing/bin/test-prompts.js b/plugins/aidd-context/hooks/routing/bin/test-prompts.js deleted file mode 100644 index e2ee569f..00000000 --- a/plugins/aidd-context/hooks/routing/bin/test-prompts.js +++ /dev/null @@ -1,98 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const fs = require('fs'); -const path = require('path'); -const { tokenize } = require('../lib/tokenize'); -const { scoreDoc } = require('../lib/bm25'); -const { dataDir } = require('../lib/paths'); - -const DATA_DIR = dataDir(); -const TESTS_FILE = path.resolve(__dirname, '..', 'tests', 'held-out-prompts.json'); -const MAX_HINTS = 3; - -function loadJson(p) { return JSON.parse(fs.readFileSync(p, 'utf8')); } - -function predict(prompt, index, thresholds) { - if (!prompt || prompt.trim().startsWith('/')) return []; - const { tokens, negated } = tokenize(prompt); - if (tokens.length === 0 || negated) return []; - const bySkill = new Map(); - for (const doc of index.scenarios) { - if (doc.polarity !== 'pos') continue; - const raw = scoreDoc(tokens, doc, index.idf, index.avgdl); - const cur = bySkill.get(doc.skill) || 0; - if (raw > cur) bySkill.set(doc.skill, raw); - } - const out = []; - for (const [skill, score] of bySkill) { - const t = thresholds[skill]; - if (!t || t.status === 'excluded' || t.tau == null) continue; - if (score >= t.tau) out.push({ skill, score }); - } - out.sort((a, b) => b.score - a.score); - return out.slice(0, MAX_HINTS); -} - -function evaluateCase(testCase, predictions) { - const matchedSkills = predictions.map(p => p.skill); - const top1 = matchedSkills[0] || null; - const accepted = [testCase.expect, ...(testCase.also_accept || [])].filter(Boolean); - - if (testCase.expect === null) { - return matchedSkills.length === 0 ? 'pass' : 'fail-fp'; - } - if (top1 === testCase.expect) return 'pass'; - if (accepted.includes(top1)) return 'pass-alt'; - if (matchedSkills.includes(testCase.expect)) return 'pass-rank'; - if (matchedSkills.length === 0) return 'fail-miss'; - return 'fail-wrong'; -} - -function main() { - const index = loadJson(path.join(DATA_DIR, 'index.json')); - const thresholds = loadJson(path.join(DATA_DIR, 'thresholds.json')); - const cases = loadJson(TESTS_FILE); - - console.log(`\nHeld-out tests (${cases.length} prompts) against build ${index.build_hash.slice(0, 8)}\n`); - console.log('result category prompt → top match (score)'); - console.log('-'.repeat(90)); - - const stats = { pass: 0, 'pass-alt': 0, 'pass-rank': 0, 'fail-fp': 0, 'fail-miss': 0, 'fail-wrong': 0 }; - const byCategory = {}; - - for (const c of cases) { - const preds = predict(c.prompt, index, thresholds); - const verdict = evaluateCase(c, preds); - stats[verdict] = (stats[verdict] || 0) + 1; - byCategory[c.category] = byCategory[c.category] || { pass: 0, fail: 0 }; - byCategory[c.category][verdict.startsWith('pass') ? 'pass' : 'fail']++; - - const top = preds[0] - ? `${preds[0].skill} (${preds[0].score.toFixed(2)})` - : '∅'; - const expect = c.expect || '∅'; - const symbol = verdict.startsWith('pass') ? '✓' : '✗'; - const promptShort = (c.prompt || '<empty>').slice(0, 28).padEnd(28); - console.log(`${symbol} ${verdict.padEnd(11)} ${c.category.padEnd(22)} "${promptShort}" → ${top} [expect ${expect}]`); - } - - console.log('\nStats:'); - console.log(` pass ${stats.pass}`); - console.log(` pass-alt ${stats['pass-alt']} (compound match)`); - console.log(` pass-rank ${stats['pass-rank']} (expected in matches but not top-1)`); - console.log(` fail-fp ${stats['fail-fp']} (matched when should be ∅)`); - console.log(` fail-miss ${stats['fail-miss']} (∅ when should match)`); - console.log(` fail-wrong ${stats['fail-wrong']} (wrong skill at top-1)`); - - const total = cases.length; - const passed = stats.pass + stats['pass-alt'] + stats['pass-rank']; - console.log(`\nOverall: ${passed}/${total} (${(passed / total * 100).toFixed(1)}%)`); - - console.log('\nBy category:'); - for (const [cat, s] of Object.entries(byCategory)) { - console.log(` ${cat.padEnd(24)} ${s.pass}/${s.pass + s.fail}`); - } -} - -main(); diff --git a/plugins/aidd-context/hooks/routing/lib/bm25.js b/plugins/aidd-context/hooks/routing/lib/bm25.js deleted file mode 100644 index c573c2d9..00000000 --- a/plugins/aidd-context/hooks/routing/lib/bm25.js +++ /dev/null @@ -1,65 +0,0 @@ -'use strict'; - -const K1 = 1.2; -const B = 0.3; - -function termFreq(tokens) { - const tf = new Map(); - for (const t of tokens) tf.set(t, (tf.get(t) || 0) + 1); - return tf; -} - -function computeIdf(docs) { - const N = docs.length; - const df = new Map(); - for (const doc of docs) { - const seen = new Set(doc.tokens); - for (const t of seen) df.set(t, (df.get(t) || 0) + 1); - } - const idf = {}; - for (const [t, n] of df) { - idf[t] = Math.log(1 + (N - n + 0.5) / (n + 0.5)); - } - return idf; -} - -function scoreDoc(queryTokens, doc, idf, avgdl) { - const tf = termFreq(doc.tokens); - const dl = doc.tokens.length || 1; - let score = 0; - for (const q of queryTokens) { - const f = tf.get(q); - if (!f) continue; - const w = idf[q] || 0; - const num = f * (K1 + 1); - const den = f + K1 * (1 - B + B * (dl / avgdl)); - score += w * (num / den); - } - return score; -} - -function buildIndex(scenarios) { - // scenarios: [{ skill, prompt, expect_action, tokens, polarity }] - const positives = scenarios.filter(s => s.polarity === 'pos'); - const avgdl = positives.reduce((a, s) => a + s.tokens.length, 0) / (positives.length || 1); - const idf = computeIdf(positives); - return { idf, avgdl, K1, B }; -} - -function scoreSkills(queryTokens, scenarios, index, options = {}) { - const { negationPenalty = 0.3, negated = false } = options; - const factor = negated ? negationPenalty : 1.0; - const bySkill = new Map(); - for (const s of scenarios) { - if (s.polarity !== 'pos') continue; - const raw = scoreDoc(queryTokens, s, index.idf, index.avgdl); - const cur = bySkill.get(s.skill) || 0; - if (raw > cur) bySkill.set(s.skill, raw); - } - const out = []; - for (const [skill, score] of bySkill) out.push({ skill, score: score * factor }); - out.sort((a, b) => b.score - a.score); - return out; -} - -module.exports = { buildIndex, scoreSkills, scoreDoc, computeIdf, K1, B }; diff --git a/plugins/aidd-context/hooks/routing/lib/paths.js b/plugins/aidd-context/hooks/routing/lib/paths.js deleted file mode 100644 index 3ce052e7..00000000 --- a/plugins/aidd-context/hooks/routing/lib/paths.js +++ /dev/null @@ -1,16 +0,0 @@ -'use strict'; - -const os = require('os'); -const path = require('path'); - -const INSTALLED_PLUGINS = path.join(os.homedir(), '.claude', 'plugins', 'installed_plugins.json'); -const KNOWN_MARKETPLACES = path.join(os.homedir(), '.claude', 'plugins', 'known_marketplaces.json'); -const USER_DEFAULT_DATA = path.join(os.homedir(), '.claude', 'plugins', 'data', 'aidd-context', 'routing'); - -function dataDir() { - if (process.env.AIDD_ROUTING_DATA) return process.env.AIDD_ROUTING_DATA; - if (process.env.CLAUDE_PLUGIN_DATA) return path.join(process.env.CLAUDE_PLUGIN_DATA, 'routing'); - return USER_DEFAULT_DATA; -} - -module.exports = { INSTALLED_PLUGINS, KNOWN_MARKETPLACES, USER_DEFAULT_DATA, dataDir }; diff --git a/plugins/aidd-context/hooks/routing/lib/tokenize.js b/plugins/aidd-context/hooks/routing/lib/tokenize.js deleted file mode 100644 index 334fbd1d..00000000 --- a/plugins/aidd-context/hooks/routing/lib/tokenize.js +++ /dev/null @@ -1,63 +0,0 @@ -'use strict'; - -const STOP_WORDS = new Set([ - 'a', 'an', 'the', 'my', 'this', 'that', 'of', 'for', 'to', 'with', 'is', - 'are', 'i', 'we', 'you', 'on', 'in', 'at', 'by', 'as', 'be', - 'what', 'whats', 'who', 'when', 'where', 'which', 'how', 'whose', - 'le', 'la', 'les', 'un', 'une', 'des', 'du', 'de', 'et', 'ou', 'je', 'tu', - 'il', 'elle', 'ce', 'cette', 'ces', 'mes', 'tes', 'ses', 'sur', 'dans', - 'moi', 'toi', 'nous', 'vous', 'que', 'qui', 'quoi', 'quand', 'comment', -]); - -const NEGATION_TOKENS = new Set([ - 'dont', 'not', 'never', 'no', - 'ne', 'pas', 'jamais', 'non', 'sans', -]); - -const SUFFIXES = [ - 'ations', 'ation', 'ements', 'ement', 'ments', 'ment', - 'tions', 'tion', 'ings', 'ing', - 'iez', 'ions', 'ais', 'ait', 'ant', 'aient', - 'ies', 'ied', 'iest', 'est', 'ers', 'er', 'ed', 'es', 's', -]; - -function accentFold(s) { - return s.normalize('NFD').replace(/[̀-ͯ]/g, ''); -} - -function stem(token) { - if (token.length <= 3) return token; - for (const suf of SUFFIXES) { - if (token.length > suf.length + 2 && token.endsWith(suf)) { - return token.slice(0, -suf.length); - } - } - return token; -} - -function tokenize(text) { - if (!text || typeof text !== 'string') return { tokens: [], negated: false }; - let s = text.replace(/([a-z])([A-Z])/g, '$1 $2'); - s = s.toLowerCase(); - s = accentFold(s); - s = s.replace(/['’]/g, ''); - s = s.replace(/[^a-z0-9\s/]/g, ' '); - const raw = s.split(/[\s_\-]+/).filter(Boolean); - let negated = false; - const tokens = []; - for (const r of raw) { - if (NEGATION_TOKENS.has(r)) { negated = true; continue; } - if (STOP_WORDS.has(r)) continue; - if (r.startsWith('/')) { - tokens.push(r); - const inner = r.slice(1); - if (inner) tokens.push(stem(inner)); - continue; - } - if (r.length === 0) continue; - tokens.push(stem(r)); - } - return { tokens, negated }; -} - -module.exports = { tokenize, stem, accentFold }; diff --git a/plugins/aidd-context/hooks/routing/routing-hint.js b/plugins/aidd-context/hooks/routing/routing-hint.js deleted file mode 100644 index 5aff0512..00000000 --- a/plugins/aidd-context/hooks/routing/routing-hint.js +++ /dev/null @@ -1,121 +0,0 @@ -#!/usr/bin/env node -'use strict'; - -const fs = require('fs'); -const path = require('path'); -const { spawn } = require('child_process'); -const { tokenize } = require('./lib/tokenize'); -const { scoreDoc } = require('./lib/bm25'); -const { dataDir, INSTALLED_PLUGINS } = require('./lib/paths'); - -const DATA_DIR = dataDir(); -const BUILD_SCRIPT = path.join(__dirname, 'bin', 'build-index.js'); -const MAX_HINTS = 3; - -function readStdin() { - return new Promise((resolve) => { - let data = ''; - process.stdin.setEncoding('utf8'); - process.stdin.on('data', (c) => { data += c; }); - process.stdin.on('end', () => resolve(data)); - process.stdin.on('error', () => resolve('')); - }); -} - -function passthrough() { process.exit(0); } - -function spawnDetachedRebuild() { - try { - const child = spawn(process.execPath, [BUILD_SCRIPT], { - detached: true, - stdio: 'ignore', - }); - child.unref(); - } catch { /* fail silent */ } -} - -function maybeAsyncRebuild() { - try { - if (!fs.existsSync(INSTALLED_PLUGINS)) return; - const manifestPath = path.join(DATA_DIR, 'manifest.json'); - if (!fs.existsSync(manifestPath)) return spawnDetachedRebuild(); - const m = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); - const installedMtime = fs.statSync(INSTALLED_PLUGINS).mtimeMs; - const builtAt = new Date(m.built_at).getTime(); - if (Number.isFinite(builtAt) && installedMtime > builtAt) spawnDetachedRebuild(); - } catch { /* fail silent */ } -} - -function loadIndex() { - try { - const index = JSON.parse(fs.readFileSync(path.join(DATA_DIR, 'index.json'), 'utf8')); - const thresholds = JSON.parse(fs.readFileSync(path.join(DATA_DIR, 'thresholds.json'), 'utf8')); - return { index, thresholds }; - } catch { - return null; - } -} - -function scoreSkills(tokens, index) { - const bySkill = new Map(); - for (const doc of index.scenarios) { - if (doc.polarity !== 'pos') continue; - const raw = scoreDoc(tokens, doc, index.idf, index.avgdl); - const cur = bySkill.get(doc.skill) || 0; - if (raw > cur) bySkill.set(doc.skill, raw); - } - const out = []; - for (const [skill, score] of bySkill) out.push({ skill, score }); - out.sort((a, b) => b.score - a.score); - return out; -} - -async function main() { - let payload = {}; - try { payload = JSON.parse(await readStdin()); } catch { return passthrough(); } - const prompt = payload.prompt; - if (!prompt || typeof prompt !== 'string') return passthrough(); - if (prompt.trim().startsWith('/')) return passthrough(); - - maybeAsyncRebuild(); - const loaded = loadIndex(); - if (!loaded) return passthrough(); - const { index, thresholds } = loaded; - - const { tokens, negated } = tokenize(prompt); - if (tokens.length === 0 || negated) return passthrough(); - - const scored = scoreSkills(tokens, index); - const filtered = scored.filter(s => { - const t = thresholds[s.skill]; - return t && t.status !== 'excluded' && t.tau != null && s.score >= t.tau; - }); - - if (filtered.length === 0) return passthrough(); - - const top = filtered.slice(0, MAX_HINTS); - const sum = top.reduce((a, b) => a + b.score, 0); - const matches = top.map(m => ({ - skill: m.skill, - score: m.score, - confidence: sum > 0 ? m.score / sum : 0, - })); - - const lines = matches - .map(m => `- ${m.skill} (${Math.round(m.confidence * 100)}% confidence, score ${m.score.toFixed(2)})`) - .join('\n'); - const note = matches.length > 1 - ? '\nMultiple candidates ranked by relative confidence. Pick the one matching user intent.' - : ''; - const text = `Routing hint (BM25, deterministic, build ${index.build_hash.slice(0, 8)}):\n${lines}${note}`; - - process.stdout.write(JSON.stringify({ - hookSpecificOutput: { - hookEventName: 'UserPromptSubmit', - additionalContext: text, - }, - })); - process.exit(0); -} - -main().catch(() => passthrough()); diff --git a/plugins/aidd-context/hooks/routing/tests/held-out-prompts.json b/plugins/aidd-context/hooks/routing/tests/held-out-prompts.json deleted file mode 100644 index 30a497ab..00000000 --- a/plugins/aidd-context/hooks/routing/tests/held-out-prompts.json +++ /dev/null @@ -1,110 +0,0 @@ -[ - { "prompt": "ship this code now", "expect": "aidd-vcs:01-commit", "category": "vcs-paraphrase-en" }, - { "prompt": "fais un commit propre", "expect": "aidd-vcs:01-commit", "category": "vcs-paraphrase-fr" }, - { "prompt": "fige cette version", "expect": "aidd-vcs:01-commit", "category": "vcs-paraphrase-fr" }, - { "prompt": "commit my changes", "expect": "aidd-vcs:01-commit", "category": "commit-en" }, - { "prompt": "save this snapshot", "expect": "aidd-vcs:01-commit", "category": "commit-en" }, - { "prompt": "valider mes modifications", "expect": "aidd-vcs:01-commit", "category": "commit-fr" }, - { "prompt": "ship this commit", "expect": "aidd-vcs:01-commit", "category": "commit-en" }, - - { "prompt": "submit a PR", "expect": "aidd-vcs:02-pull-request", "category": "vcs-paraphrase-en" }, - { "prompt": "demande une revue de code", "expect": "aidd-vcs:02-pull-request", "category": "vcs-paraphrase-fr" }, - { "prompt": "open a pull request", "expect": "aidd-vcs:02-pull-request", "category": "pr-en" }, - { "prompt": "create merge request", "expect": "aidd-vcs:02-pull-request", "category": "pr-en" }, - { "prompt": "ouvre une PR", "expect": "aidd-vcs:02-pull-request", "category": "pr-fr" }, - { "prompt": "submit PR for review", "expect": "aidd-vcs:02-pull-request", "category": "pr-en" }, - { "prompt": "draft pull request", "expect": "aidd-vcs:02-pull-request", "category": "pr-en" }, - - { "prompt": "publish version 5.0", "expect": "aidd-vcs:03-release-tag", "category": "vcs-paraphrase-en" }, - { "prompt": "fais une release", "expect": "aidd-vcs:03-release-tag", "category": "vcs-paraphrase-fr" }, - { "prompt": "tag v3.5", "expect": "aidd-vcs:03-release-tag", "category": "release-en" }, - { "prompt": "cut release v4.0", "expect": "aidd-vcs:03-release-tag", "category": "release-en" }, - { "prompt": "fais une release v3", "expect": "aidd-vcs:03-release-tag", "category": "release-fr" }, - { "prompt": "bump version major", "expect": "aidd-vcs:03-release-tag", "category": "release-en" }, - - { "prompt": "report a bug", "expect": "aidd-vcs:04-issue-create", "category": "issue-en" }, - { "prompt": "file an issue for login crash", "expect": "aidd-vcs:04-issue-create", "category": "issue-en" }, - { "prompt": "déclare un bug", "expect": "aidd-vcs:04-issue-create", "category": "issue-fr" }, - - { "prompt": "show ticket ABC-123", "expect": "aidd-pm:01-ticket-info", "category": "ticket-en" }, - { "prompt": "what's ENG-42 about", "expect": "aidd-pm:01-ticket-info", "category": "ticket-en" }, - { "prompt": "détails du ticket JIRA", "expect": "aidd-pm:01-ticket-info", "category": "ticket-fr" }, - - { "prompt": "create user stories for checkout", "expect": "aidd-pm:02-user-stories-create", "category": "stories-en" }, - { "prompt": "INVEST stories for billing", "expect": "aidd-pm:02-user-stories-create", "category": "stories-en" }, - { "prompt": "rédige user stories pour login", "expect": "aidd-pm:02-user-stories-create", "category": "stories-fr" }, - - { "prompt": "draft a PRD for search", "expect": "aidd-pm:03-prd", "category": "prd-en" }, - { "prompt": "write product requirements", "expect": "aidd-pm:03-prd", "category": "prd-en" }, - { "prompt": "génère un PRD", "expect": "aidd-pm:03-prd", "category": "prd-fr" }, - - { "prompt": "build spec from PRD", "expect": "aidd-pm:04-spec", "category": "spec-en", "also_accept": ["aidd-pm:03-prd"] }, - { "prompt": "refine spec with reviewer notes", "expect": "aidd-pm:04-spec", "category": "spec-en" }, - { "prompt": "rédige la spec", "expect": "aidd-pm:04-spec", "category": "spec-fr" }, - - { "prompt": "create implementation plan for auth", "expect": "aidd-dev:01-plan", "category": "plan-en" }, - { "prompt": "rédige un plan technique", "expect": "aidd-dev:01-plan", "category": "plan-fr" }, - { "prompt": "design phase by phase", "expect": "aidd-dev:01-plan", "category": "plan-en", "also_accept": ["aidd-dev:02-implement"] }, - - { "prompt": "implement the plan", "expect": "aidd-dev:02-implement", "category": "implement-en" }, - { "prompt": "execute plan from aidd_docs", "expect": "aidd-dev:02-implement", "category": "implement-en" }, - { "prompt": "exécute le plan", "expect": "aidd-dev:02-implement", "category": "implement-fr" }, - - { "prompt": "assert feature works", "expect": "aidd-dev:03-assert", "category": "assert-en" }, - { "prompt": "verify implementation", "expect": "aidd-dev:03-assert", "category": "assert-en", "also_accept": ["aidd-dev:05-review"] }, - - { "prompt": "audit codebase", "expect": "aidd-dev:04-audit", "category": "audit-en" }, - { "prompt": "find tech debt", "expect": "aidd-dev:04-audit", "category": "audit-en" }, - - { "prompt": "review code", "expect": "aidd-dev:05-review", "category": "review-en" }, - { "prompt": "code review against rules", "expect": "aidd-dev:05-review", "category": "review-en" }, - - { "prompt": "write tests", "expect": "aidd-dev:06-test", "category": "test-en" }, - { "prompt": "test user journey", "expect": "aidd-dev:06-test", "category": "test-en" }, - - { "prompt": "OWASP fix", "expect": "aidd-dev:07-refactor", "category": "refactor-en" }, - { "prompt": "optimize performance", "expect": "aidd-dev:07-refactor", "category": "refactor-en" }, - - { "prompt": "reproduce bug", "expect": "aidd-dev:08-debug", "category": "debug-en" }, - { "prompt": "debug payment crash", "expect": "aidd-dev:08-debug", "category": "debug-en" }, - - { "prompt": "loop until tests pass", "expect": "aidd-dev:09-for-sure", "category": "for-sure-en" }, - { "prompt": "for sure fix this", "expect": "aidd-dev:09-for-sure", "category": "for-sure-en" }, - - { "prompt": "onboard me", "expect": "aidd-context:00-onboard", "category": "onboard-en" }, - { "prompt": "aidd init", "expect": "aidd-context:02-project-init", "category": "init-en" }, - { "prompt": "generate mermaid", "expect": "aidd-context:04-mermaid", "category": "mermaid-en" }, - - { "prompt": "challenge this", "expect": "aidd-refine:02-challenge", "category": "challenge-en" }, - { "prompt": "brainstorm", "expect": "aidd-refine:01-brainstorm", "category": "brainstorm-en" }, - - { "prompt": "setup async dev", "expect": "aidd-orchestrator:00-async-dev", "category": "orch-setup-en" }, - { "prompt": "run async pipeline", "expect": "aidd-orchestrator:00-async-dev", "category": "orch-run-en" }, - { "prompt": "handle review feedback", "expect": "aidd-orchestrator:00-async-dev", "category": "orch-review-en" }, - - { "prompt": "list all skills", "expect": "aidd-context:06-discovery", "category": "discovery" }, - { "prompt": "what hooks are installed", "expect": "aidd-context:06-discovery", "category": "discovery" }, - { "prompt": "show me available agents", "expect": "aidd-context:06-discovery", "category": "discovery" }, - { "prompt": "quels skills sont dispos", "expect": "aidd-context:06-discovery", "category": "discovery-fr" }, - { "prompt": "trouve la règle pour les API", "expect": "aidd-context:06-discovery", "category": "discovery-fr" }, - - { "prompt": "what's the weather today", "expect": null, "category": "out-of-scope" }, - { "prompt": "explique-moi le code", "expect": null, "category": "out-of-scope" }, - { "prompt": "hello", "expect": null, "category": "out-of-scope" }, - { "prompt": "what's the time", "expect": null, "category": "out-of-scope" }, - { "prompt": "tell me a joke", "expect": null, "category": "out-of-scope" }, - - { "prompt": "don't commit anything", "expect": null, "category": "negation" }, - { "prompt": "ne fais pas de PR", "expect": null, "category": "negation-fr" }, - { "prompt": "never tag this", "expect": null, "category": "negation" }, - { "prompt": "ne fais pas de release", "expect": null, "category": "negation-fr" }, - - { "prompt": "/commit and push", "expect": null, "category": "slash-passthrough" }, - { "prompt": "/commit", "expect": null, "category": "slash-passthrough" }, - - { "prompt": "", "expect": null, "category": "empty" }, - - { "prompt": "commit and open pr", "expect": "aidd-vcs:01-commit", "category": "compound-multi", "also_accept": ["aidd-vcs:02-pull-request"] }, - { "prompt": "save and tag v2", "expect": "aidd-vcs:01-commit", "category": "compound-multi", "also_accept": ["aidd-vcs:03-release-tag"] }, - { "prompt": "commit and open PR", "expect": "aidd-vcs:01-commit", "category": "compound-multi", "also_accept": ["aidd-vcs:02-pull-request"] } -] diff --git a/plugins/aidd-context/skills/00-onboard/evals/scenarios.json b/plugins/aidd-context/skills/00-onboard/evals/scenarios.json deleted file mode 100644 index 0c0aa7b4..00000000 --- a/plugins/aidd-context/skills/00-onboard/evals/scenarios.json +++ /dev/null @@ -1,18 +0,0 @@ -[ - { "prompt": "Where do I start with this project?", "expect_action": "detect-state" }, - { "prompt": "Onboard me to this project", "expect_action": "detect-state" }, - { "prompt": "What should I work on next?", "expect_action": "detect-state" }, - { "prompt": "What should I run next?", "expect_action": "detect-state" }, - { "prompt": "What's the current state of this project?", "expect_action": "detect-state" }, - { "prompt": "Guide me through the AIDD journey", "expect_action": "detect-state" }, - { "prompt": "Guide me through aidd-context", "expect_action": "detect-state" }, - { "prompt": "I'm new here, onboard me", "expect_action": "detect-state" }, - { "prompt": "Run /aidd-context:00-onboard", "expect_action": "detect-state" }, - { "prompt": "Présente-moi l'état du projet et la prochaine étape", "expect_action": "detect-state" }, - { "prompt": "Par où je commence sur ce projet", "expect_action": "detect-state" }, - { "prompt": "Quelle est la prochaine étape AIDD", "expect_action": "detect-state" }, - { "prompt": "Edit step 3 of actions/01-detect-state.md", "expect_action": null }, - { "prompt": "List every installed skill so I can pick one", "expect_action": null }, - { "prompt": "Generate a Mermaid diagram for the auth flow", "expect_action": null }, - { "prompt": "Update memory bank with our new ADR", "expect_action": null } -] diff --git a/plugins/aidd-context/skills/01-bootstrap/evals/scenarios.json b/plugins/aidd-context/skills/01-bootstrap/evals/scenarios.json deleted file mode 100644 index db495899..00000000 --- a/plugins/aidd-context/skills/01-bootstrap/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "I want to bootstrap a new SaaS for managing client invoices", "expect_action": "gather-needs" }, - { "prompt": "Help me imagine the technical architecture for my project", "expect_action": "gather-needs" }, - { "prompt": "Generate INSTALL.md for my project", "expect_action": "gather-needs" }, - { "prompt": "Bootstrap a new SaaS project from scratch", "expect_action": "gather-needs" }, - { "prompt": "Choose a stack for my new SaaS", "expect_action": "gather-needs" }, - { "prompt": "Monolith vs microservices for my SaaS startup", "expect_action": "gather-needs" }, - { "prompt": "Validate the technical stack of my SaaS", "expect_action": "gather-needs" }, - { "prompt": "Bootstrap saas architecture from scratch", "expect_action": "gather-needs" }, - { "prompt": "Démarre une SaaS et conçois l'architecture", "expect_action": "gather-needs" }, - { "prompt": "Aide-moi à imaginer l'architecture de ma SaaS", "expect_action": "gather-needs" }, - { "prompt": "Génère le INSTALL.md de mon nouveau projet SaaS", "expect_action": "gather-needs" }, - { "prompt": "Valide mon stack SaaS monolith vs microservices", "expect_action": "gather-needs" }, - { "prompt": "Design my database schema for users and orders", "expect_action": null }, - { "prompt": "Set up my CI/CD pipeline", "expect_action": null }, - { "prompt": "Refactor my React component", "expect_action": null } -] diff --git a/plugins/aidd-context/skills/02-project-init/assets/CONTRIBUTING.md b/plugins/aidd-context/skills/02-project-init/assets/CONTRIBUTING.md index a356b89f..dab102ed 100644 --- a/plugins/aidd-context/skills/02-project-init/assets/CONTRIBUTING.md +++ b/plugins/aidd-context/skills/02-project-init/assets/CONTRIBUTING.md @@ -8,7 +8,7 @@ Use the generator skills to scaffold new content that follows the framework stru | Skill | Creates | | ---------------------------------- | -------------------- | -| `aidd-context:03-context-generate` | New skill, agent, or rule (router-based, with actions and evals) | +| `aidd-context:03-context-generate` | New skill, agent, or rule (router-based, with actions) | | `aidd-context:05-learn` | New memory or rule capturing a learning | Generator skills consume the templates inside their `assets/` folder and write the output to the correct location for your AI tool (Claude Code, Cursor, Copilot, Codex, OpenCode). @@ -19,7 +19,7 @@ All templates live alongside the skill that owns them, under `plugins/<plugin>/s | Where | What it scaffolds | | -------------------------------------------------- | -------------------------------------------------------- | -| `aidd-context:03-context-generate/assets/skills/` | `SKILL.md`, action, evals templates | +| `aidd-context:03-context-generate/assets/skills/` | `SKILL.md`, action templates | | `aidd-context:03-context-generate/assets/agents/` | Agent file template | | `aidd-context:03-context-generate/assets/rules/` | Rule file template | | `aidd-pm:03-prd/assets/` | PRD body template | @@ -38,7 +38,6 @@ When tools differ in syntax (frontmatter, slash command name, references), follo - Open a pull request for any new skill, agent, rule, or template. Visible changes that affect how the AI behaves on the project deserve team review. - Keep skills router-pure: SKILL.md holds no business logic; everything lives inside actions. -- Add evals (`evals/scenarios.json`) for every auto-trigger skill so router behavior stays correct over time. - Stay within 5 to 10 percent deviation from a template structure. Beyond that, update the template first, then derive the new content from it. ## Conventions diff --git a/plugins/aidd-context/skills/02-project-init/assets/GUIDELINES.md b/plugins/aidd-context/skills/02-project-init/assets/GUIDELINES.md index 96b306ef..489190f5 100644 --- a/plugins/aidd-context/skills/02-project-init/assets/GUIDELINES.md +++ b/plugins/aidd-context/skills/02-project-init/assets/GUIDELINES.md @@ -124,5 +124,4 @@ For each increment: - Anthropic, "Prompt engineering overview": <https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview> - Anthropic, "Claude Code memory": <https://docs.claude.com/en/docs/claude-code/memory> - OpenAI, "Prompt engineering best practices": <https://help.openai.com/en/articles/6654000-best-practices-for-prompting> -- OpenAI Cookbook, "Eval-driven development": <https://cookbook.openai.com/examples/evaluation/use-cases/evalsapi_tools_evaluation> - GitHub Docs, "Repository custom instructions for Copilot": <https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/adding-repository-custom-instructions-for-github-copilot> diff --git a/plugins/aidd-context/skills/02-project-init/assets/README.md b/plugins/aidd-context/skills/02-project-init/assets/README.md index 7db06add..2f11741f 100644 --- a/plugins/aidd-context/skills/02-project-init/assets/README.md +++ b/plugins/aidd-context/skills/02-project-init/assets/README.md @@ -138,7 +138,6 @@ Inside the synchronous path, `aidd-dev:00-sdlc` is the Dev SDLC orchestrator tha - Skills must have an `## Available actions` table, `## Default flow`, `## Transversal rules`. - Actions must contain only `## Inputs`, `## Outputs`, `## Process`, `## Test`. - Tests must be observable: command, artifact check, or side effect. -- Evals (`evals/scenarios.json`) ship for every auto-trigger skill. --- diff --git a/plugins/aidd-context/skills/02-project-init/evals/scenarios.json b/plugins/aidd-context/skills/02-project-init/evals/scenarios.json deleted file mode 100644 index 79da67ee..00000000 --- a/plugins/aidd-context/skills/02-project-init/evals/scenarios.json +++ /dev/null @@ -1,18 +0,0 @@ -[ - { "prompt": "aidd init", "expect_action": "init-context-file" }, - { "prompt": "Initialize the project memory bank", "expect_action": "init-context-file" }, - { "prompt": "Bootstrap this repo with AIDD context", "expect_action": "init-context-file" }, - { "prompt": "Set up aidd_docs and rules scaffold", "expect_action": "scaffold-docs" }, - { "prompt": "Refresh the memory bank for this project", "expect_action": "generate-memory" }, - { "prompt": "Run aidd init for the first time", "expect_action": "init-context-file" }, - { "prompt": "Initialize aidd_docs structure", "expect_action": "scaffold-docs" }, - { "prompt": "Set up project memory bank initial scaffold", "expect_action": "scaffold-docs" }, - { "prompt": "Generate the memory bank for my project", "expect_action": "generate-memory" }, - { "prompt": "Initialise aidd dans ce projet", "expect_action": "init-context-file" }, - { "prompt": "Crée le memory bank du projet", "expect_action": "scaffold-docs" }, - { "prompt": "Génère la memory bank du projet", "expect_action": "generate-memory" }, - { "prompt": "Configure aidd pour mon repo", "expect_action": "init-context-file" }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Update a single memory file", "expect_action": null }, - { "prompt": "Edit a rule file", "expect_action": null } -] diff --git a/plugins/aidd-context/skills/03-context-generate/README.md b/plugins/aidd-context/skills/03-context-generate/README.md index c9a79b8c..58225257 100644 --- a/plugins/aidd-context/skills/03-context-generate/README.md +++ b/plugins/aidd-context/skills/03-context-generate/README.md @@ -4,7 +4,7 @@ Generates the seven context artifacts a project can consume, across the host AI tool(s) detected in the project. Before writing any artifact the skill runs the Model Y gate: detect installed tools from D1 signals, propose the set to the user, wait for explicit confirmation (1..N), then for each (artifact, confirmed tool) look up `references/ai-mapping.md`; if unsupported, block with explanation (D2) and continue the rest. -- **Skills** - router-based: `SKILL.md` router + atomic testable actions + minimal evals. +- **Skills** - router-based: `SKILL.md` router + atomic testable actions. - **Agents** - single-file agent definitions following the framework's agent template. - **Rules** - framework rule files governing editor / agent behaviour. - **Commands** - flat `.md` slash command files (frontmatter + body), for one-shot manual triggers without supporting files. @@ -12,7 +12,7 @@ Generates the seven context artifacts a project can consume, across the host AI - **Plugins** - full plugin scaffold (a plugin manifest + README + slot dirs, path resolved per tool from `references/ai-mapping.md`; optional seed skill). - **Marketplaces** - a marketplace catalog file (path resolved per tool from `references/ai-mapping.md`) that distributes one or more plugins. -Evaluations are declared before implementation; every action carries a `## Test`. +Every action carries a `## Test`. ## When to use @@ -33,14 +33,13 @@ Evaluations are declared before implementation; every action carries a `## Test` Use skill aidd-context:03-context-generate ``` -For skill generation, the skill walks 6 atomic actions: +For skill generation, the skill walks 5 atomic actions: 1. `capture-intent` - clarify the desired output and decide generate-vs-modify. -2. `design-evals` - write a minimal `scenarios.json`. -3. `decompose-actions` - list actions and their `## Test` sentences. -4. `draft-skill` - write the `SKILL.md` router. -5. `write-actions` - write each action file. -6. `validate` - spawn one agent per action, run its `## Test`, and aggregate into a pass/fail report. +2. `decompose-actions` - list actions and their `## Test` sentences. +3. `draft-skill` - write the `SKILL.md` router. +4. `write-actions` - write each action file. +5. `validate` - spawn one agent per action, run its `## Test`, and aggregate into a pass/fail report. The other six artifact types have their own sub-flows under `actions/<sub-domain>/`. Each entry action runs the Model Y tool-resolution gate (detect -> propose -> confirm -> D2 block) before writing. @@ -53,7 +52,7 @@ The other six artifact types have their own sub-flows under `actions/<sub-domain ## Outputs -- A new or refactored skill directory: `SKILL.md` + `actions/*.md` + `evals/scenarios.json` (when auto-triggered) + optional `references/` and `assets/`. +- A new or refactored skill directory: `SKILL.md` + `actions/*.md` + optional `references/` and `assets/`. - Or a generated agent file from `assets/agents/agent-template.md`. - Or a generated rule file from `assets/rules/rule-template.md`. - Or a flat slash command file from `assets/commands/command-template.md`. @@ -68,8 +67,8 @@ The other six artifact types have their own sub-flows under `actions/<sub-domain ## Rules -R1-R11 in [`SKILL.md`](SKILL.md) are the non-bypassable invariants: -`SKILL.md` is a pure router, one skill = one domain, references one-level deep, `SKILL.md` ≤ 500 lines, descriptions must include explicit triggers and a "Do NOT use for..." clause, every action has a `## Test`, auto-trigger skills ship at least 3 eval scenarios, and the tool-resolution gate (detect -> propose -> confirm -> D2 block) runs before writing any artifact in generate mode. +The invariants in [`SKILL.md`](SKILL.md) (R1-R9 plus the R10 tool-resolution gate) are non-bypassable: +`SKILL.md` is a pure router, one skill = one domain, references one-level deep, `SKILL.md` ≤ 500 lines, descriptions must include explicit triggers and a "Do NOT use for..." clause, every action has a `## Test`, and the tool-resolution gate (detect -> propose -> confirm -> D2 block) runs before writing any artifact in generate mode. ## Technical details diff --git a/plugins/aidd-context/skills/03-context-generate/SKILL.md b/plugins/aidd-context/skills/03-context-generate/SKILL.md index 0df0944f..22c1f555 100644 --- a/plugins/aidd-context/skills/03-context-generate/SKILL.md +++ b/plugins/aidd-context/skills/03-context-generate/SKILL.md @@ -8,7 +8,7 @@ description: Generate context artifacts (skills, agents, rules, commands, hooks, Generates the seven context artifacts a project consumes, with per-tool path resolution. -- **Skills** - router-based: `SKILL.md` router + atomic testable actions + minimal evals. +- **Skills** - router-based: `SKILL.md` router + atomic testable actions. - **Agents** - single-file agent definitions following the framework's agent template. - **Rules** - framework rule files governing editor/agent behavior. - **Commands** - flat `.md` slash command files (frontmatter + body), for one-shot manual triggers. @@ -22,7 +22,7 @@ Each artifact type has its own sub-flow under `@actions/<sub-domain>/`. All sub- | Sub-domain | Actions count | Entry action | Flow type | | -------------- | ------------- | ------------------------------------------------------------------------ | ------------------- | -| `skills` | 6 | `@actions/skills/01-capture-intent.md` | sequential 01..06 | +| `skills` | 5 | `@actions/skills/01-capture-intent.md` | sequential 01..05 | | `agents` | 1 | `@actions/agents/01-generate-agent.md` | single action | | `rules` | 1 | `@actions/rules/01-generate-rules.md` | single action | | `commands` | 1 | `@actions/commands/01-generate-command.md` | single action | @@ -32,7 +32,7 @@ Each artifact type has its own sub-flow under `@actions/<sub-domain>/`. All sub- ## Default flow -For sequential sub-flows, run actions in order. After each action, run its `## Test` before moving to the next. In the `skills` sub-flow, action 02 self-skips when `01` outputs `invocation_mode = manual`. +For sequential sub-flows, run actions in order. After each action, run its `## Test` before moving to the next. ## Modify flow @@ -47,14 +47,14 @@ Materialize the sub-flow as a task list at skill entry; a task closes only when ## Transversal rules - Writes are CWD-relative; the plugin install root is for reading templates only, never for prefixing a write target. -- Skills sub-flow applies R1-R10 from `references/skill-authoring.md` to every generated skill. Other sub-flows follow their own conventions in `assets/` and `references/`. -- R11 - Tool resolution gate (generate-only): detect, propose, confirm 1..N, then look up `references/ai-mapping.md` per (artifact, tool); block unsupported pairs (D2) and continue the rest. Skip the gate in modify mode (tool fixed by the existing artifact's on-disk location). Procedure: `references/tool-resolution.md`. +- Skills sub-flow applies R1-R9 from `references/skill-authoring.md` to every generated skill. Other sub-flows follow their own conventions in `assets/` and `references/`. +- R10 - Tool resolution gate (generate-only): detect, propose, confirm 1..N, then look up `references/ai-mapping.md` per (artifact, tool); block unsupported pairs (D2) and continue the rest. Skip the gate in modify mode (tool fixed by the existing artifact's on-disk location). Procedure: `references/tool-resolution.md`. ## References - `@references/tool-resolution.md` - shared detect/propose/confirm/D2 procedure (called by every entry action) - `@references/ai-mapping.md` - per-tool paths, frontmatter reconciliation, hooks/plugins/marketplaces map, event casing, validator commands -- `@references/skill-authoring.md` - generated-skill authoring: R1-R10 rules, anatomy, naming +- `@references/skill-authoring.md` - generated-skill authoring: R1-R9 rules, anatomy, naming - `@references/command.md` - generated-command authoring: forms, scopes, frontmatter, phases, arguments - `@references/rule.md` - generated-rule authoring: naming, category taxonomy, content format - `@references/hook.md` - Claude Code hooks deep reference: handler types, events, exit codes @@ -66,7 +66,6 @@ Skills templates (cross-tool, shared): - `@assets/skills/skill-template.md` - SKILL.md skeleton - `@assets/skills/action-template.md` - action file skeleton -- `@assets/skills/evals-template.md` - `scenarios.json` minimal schema Per-tool agent templates: diff --git a/plugins/aidd-context/skills/03-context-generate/actions/plugins/03-seed-first-skill.md b/plugins/aidd-context/skills/03-context-generate/actions/plugins/03-seed-first-skill.md index 89b62462..f5224520 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/plugins/03-seed-first-skill.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/plugins/03-seed-first-skill.md @@ -26,13 +26,12 @@ seed_skill_status: created | skipped 2. Set working location to `<plugins-root>/<plugin_name>/skills/` with skill prefix pinned to `01-`. 3. Run the skill-generation flow against that location, in order, passing `skill_name = seed_skill.name` and `expected_output = seed_skill.description`: - `@../skills/01-capture-intent.md` - - `@../skills/02-design-evals.md` - - `@../skills/03-decompose-actions.md` - - `@../skills/04-draft-skill.md` - - `@../skills/05-write-actions.md` - - `@../skills/06-validate.md` + - `@../skills/02-decompose-actions.md` + - `@../skills/03-draft-skill.md` + - `@../skills/04-write-actions.md` + - `@../skills/05-validate.md` 4. Each upstream action's `## Test` must pass before the next is invoked, per the skill flow's own contract. -5. Record `seed_skill_status = created` once `@../skills/06-validate.md` returns a green table. +5. Record `seed_skill_status = created` once `@../skills/05-validate.md` returns a green table. ## Test diff --git a/plugins/aidd-context/skills/03-context-generate/actions/plugins/04-validate.md b/plugins/aidd-context/skills/03-context-generate/actions/plugins/04-validate.md index d047a7e8..8748f3f0 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/plugins/04-validate.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/plugins/04-validate.md @@ -38,7 +38,7 @@ Resolve each tool's validator command and no-validator fallback from the **Plugi 3. **Name parity.** For each tool: `plugin.json` `name` matches the directory name and matches the `<plugin>` segment used in every nested skill frontmatter `name:`. 4. **Artifact-set parity.** For each slot enabled in `artifact_set`, at least one artifact file exists (or a `.gitkeep` is present and intentional). For each slot disabled, the corresponding subdir is absent. - Special case for `commands/`: if the tool is Codex CLI, the `commands/` subdir must be ABSENT regardless of `artifact_set.commands` value (scaffold skipped it per the Codex D2 rule). Record a warning finding if `commands/` exists under a Codex CLI plugin tree. -5. **Seed-skill structure.** If a seed skill was created, run `@../skills/06-validate.md` against `<plugins-root>/<plugin_name>/skills/01-<seed_skill.name>/`; any `❌` row downgrades `validation_status` to `fail`. +5. **Seed-skill structure.** If a seed skill was created, run `@../skills/05-validate.md` against `<plugins-root>/<plugin_name>/skills/01-<seed_skill.name>/`; any `❌` row downgrades `validation_status` to `fail`. 6. **Plugin reload smoke.** For Claude Code only: when the host runtime supports it, invoke `/reload-plugins` and confirm the new plugin appears in the load report. Skip with a warning otherwise. 7. Set `validation_status = pass` iff no `error`-severity finding was recorded across all confirmed tools. diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/01-capture-intent.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/01-capture-intent.md index ced85e79..ad07e8b6 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/01-capture-intent.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/skills/01-capture-intent.md @@ -35,14 +35,14 @@ blocked_tools: ## Process -Skill-generation rules (R1-R10) are in `@../../references/skill-authoring.md`. All steps below MUST comply with those rules when generating a new skill. +Skill-generation rules (R1-R9) are in `@../../references/skill-authoring.md`. All steps below MUST comply with those rules when generating a new skill. 1. Apply the **asset-access precheck** from `@../../references/tool-resolution.md` (## Asset access precheck). 2. Apply the **target scope selection** from `@../../references/tool-resolution.md` (## Target scope selection). 3. Ask: **generate** a new skill or **modify** an existing one? 4. Inventory project + global skills across all AI tools' skills roots (resolved from `@../../references/ai-mapping.md`; paths are CWD-relative, scan them directly from the workspace root). Read each `SKILL.md` frontmatter (`name`, first line of `description`). Print as a markdown table. 5. Branch: - - `modify` -- confirm target name exists, read its `SKILL.md`, jump to action 03. (Generate-only gate does not apply in modify mode.) + - `modify` -- confirm target name exists, read its `SKILL.md`, jump to action 02. (Generate-only gate does not apply in modify mode.) - `generate` -- ask the skill's single purpose in one sentence. If multiple unrelated domains, propose a split. 6. Pick `domain_type` (tool/activity), validate `skill_name` per `references/skill-authoring.md`. 7. Surface overlaps: same name -- block; trigger/MCP overlap with another skill -- ask merge / rename / scope-tighten / abort. Cross-skill dependency -- declare it for the SKILL.md "External data" section. @@ -52,7 +52,7 @@ Skill-generation rules (R1-R10) are in `@../../references/skill-authoring.md`. A - If the user describes a **persistent convention** (always apply, never auto-triggered), surface the rule option (`actions/rules/01-generate-rules.md`). - If the artifact has **only 1 atomic action**, present both options to the user with their trade-offs: - Flat command (`actions/commands/01-generate-command.md`) - lighter, single `.md`, no folder overhead. Right when no supporting files needed. - - Skill - heavier shell, but supports `assets/`, `references/`, `evals/`, `scripts/`. Right when even one action benefits from templates, fixtures, secrets, or test scenarios. + - Skill - heavier shell, but supports `assets/`, `references/`, `scripts/`. Right when even one action benefits from templates, fixtures, or secrets. - **Let the user decide.** Honor an explicit "skill" choice even for a 1-action artifact. 10. **Resolve target tools.** Follow `@../../references/tool-resolution.md` (detect, propose, confirm 1..N). For each confirmed tool, look up the skills surface in `@../../references/ai-mapping.md`; if the cell is marked unsupported, apply the D2 block for that tool and record it in `blocked_tools`. Continue with the remaining supported tools. diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/03-decompose-actions.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/02-decompose-actions.md similarity index 80% rename from plugins/aidd-context/skills/03-context-generate/actions/skills/03-decompose-actions.md rename to plugins/aidd-context/skills/03-context-generate/actions/skills/02-decompose-actions.md index 01f8a5e7..b4e6b57a 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/03-decompose-actions.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/skills/02-decompose-actions.md @@ -1,11 +1,11 @@ -# 03 - Decompose into actions +# 02 - Decompose into actions Break the skill into atomic, testable actions - one action, one unambiguous job. ## Inputs - `expected_output` (from 01) -- `evals/scenarios.json` (from 02 in generate flow, or existing in modify flow) +- `skill_name`, `domain_type`, `sequential` (from 01) ## Outputs @@ -17,13 +17,13 @@ An `action_plan` table. Example for a hypothetical `slack` skill: | `get-history` | Fetch the last N messages of a channel | Fetch last 5 from #general; assert array length = 5 and each entry has `ts` + `user` fields. | - | | `create-channel` | Create a channel (name, is_private) → channel_id | Create `#test-{ts}`; assert `channel_id` returned + channel listed via `slack_list_channels`. | - | -The `test` cell of each row will be **transcribed verbatim** into the `## Test` section of the generated action file in 05. No transformation. Concrete inputs, concrete assertions, observable side-effect. +The `test` cell of each row will be **transcribed verbatim** into the `## Test` section of the generated action file in 04. No transformation. Concrete inputs, concrete assertions, observable side-effect. Tests must be real-execution: status 200, artifact created, MCP returning the expected value. Never a mocked `*.test.js` - the first successful run is the test. ## Process -1. For each `expect_action` in evals (excluding `null`), trace backward: what action produces that output? What feeds it? +1. Starting from `expected_output`, trace backward: what action produces that output? What feeds it? Enumerate every distinct job the skill must perform to reach it. 2. Group by atomicity. Split if process > ~100 lines. Merge + parameterize if ≥ 80% logic shared. 3. One-shot configs (API key load, `.env` source, client init) stay inline in the consuming action's `## Process`. Create a dedicated action only if independently callable OR reused by ≥ 2 downstream actions. 4. Ordering: `sequential = true` → numbered prefixes `01-`, `02-` (see `references/skill-authoring.md` ## Naming). @@ -32,4 +32,4 @@ Tests must be real-execution: status 200, artifact created, MCP returning the ex ## Test -Every `expect_action` from evals (excluding `null`) appears in the table exactly once; every row has a non-empty `test` cell explicitly approved by the user in writing; no row depends on a downstream slug. +Every distinct job required to produce `expected_output` appears in the table exactly once; every row has a non-empty `test` cell explicitly approved by the user in writing; no row depends on a downstream slug. diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/02-design-evals.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/02-design-evals.md deleted file mode 100644 index f0316668..00000000 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/02-design-evals.md +++ /dev/null @@ -1,47 +0,0 @@ -# 02 - Design evaluations - -Write `evals/scenarios.json` so we can probe trigger correctness later. - -**Skip condition:** if `invocation_mode = manual` (from 01), skip - manual-only skills have no autonomous trigger. Jump to action 03. - -## Inputs - -- `invocation_mode` (from 01) - must be `auto`; otherwise skip this action -- `skill_name`, `expected_output` (from 01) -- `confirmed_tools`, `blocked_tools` (from 01) -- `target_base` (from 01). Empty string means project root; `plugins/<plugin-name>/` means write under that plugin. -- 3+ realistic user prompts - -## Outputs - -One `evals/scenarios.json` per confirmed tool, written to that tool's skill root - see `@../../assets/skills/evals-template.md` for schema and example. - -```yaml -files_written: - - { tool: <id>, path: <target_base><tool skills root>/<skill_name>/evals/scenarios.json } -``` - -## Process - -1. Ask the user for 3+ realistic prompts (verbatim, not invented). -2. For each prompt, map to an `expect_action` slug - or `null` if the skill must NOT trigger. -3. For each confirmed tool (skip any in `blocked_tools`), resolve the tool skills root from `@../../references/ai-mapping.md`. Write `evals/scenarios.json` to `<target_base><tool skills root>/<skill_name>/evals/scenarios.json`. If `target_base` is empty, the path is CWD-relative (e.g. `.claude/skills/<skill_name>/evals/scenarios.json`); if non-empty, prepend it (e.g. `plugins/my-plugin/.claude/skills/<skill_name>/evals/scenarios.json`). -4. Read scenarios back to the user. Wait for written validation before action 03. -5. Apply the **write target validation** from `@../../references/tool-resolution.md` (## Write target validation). - -## Test - -```bash -# Test: each written evals/scenarios.json exists, parses as JSON array with >= 3 entries -# files_written is an array of paths emitted by Process step 3 -for path in "${files_written[@]}"; do - test -f "$path" || exit 1 - node -e "const a=require('./${path}'); if (!Array.isArray(a)||a.length<3) process.exit(1);" || exit 1 - node -e "const a=require('./${path}'); a.forEach(e=>{if(typeof e.prompt!=='string') process.exit(1);});" || exit 1 - # When target_base is non-empty, assert path starts with target_base - if [ -n "${target_base}" ]; then - [[ "$path" == "${target_base}"* ]] || exit 1 - fi -done -echo ok -``` diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/04-draft-skill.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/03-draft-skill.md similarity index 93% rename from plugins/aidd-context/skills/03-context-generate/actions/skills/04-draft-skill.md rename to plugins/aidd-context/skills/03-context-generate/actions/skills/03-draft-skill.md index 4e1777cb..a0905a32 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/04-draft-skill.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/skills/03-draft-skill.md @@ -1,4 +1,4 @@ -# 04 - Draft SKILL.md router +# 03 - Draft SKILL.md router Write the SKILL.md. Router only - no business logic. @@ -7,7 +7,7 @@ Write the SKILL.md. Router only - no business logic. - `skill_name`, `domain_type`, `expected_output`, `invocation_mode` (from 01) - `confirmed_tools`, `blocked_tools` (from 01) - `target_base` (from 01). Empty string means project root; `plugins/<plugin-name>/` means write under that plugin. -- `action_plan` (from 03) +- `action_plan` (from 02) ## Outputs @@ -47,4 +47,4 @@ done echo ok ``` -Quality: action table slugs match `action_plan` from 03; every non-null `expect_action` in `evals/scenarios.json` matches a slug in the action table (manual check). +Quality: action table slugs match `action_plan` from 02 (manual check). diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/05-write-actions.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/04-write-actions.md similarity index 96% rename from plugins/aidd-context/skills/03-context-generate/actions/skills/05-write-actions.md rename to plugins/aidd-context/skills/03-context-generate/actions/skills/04-write-actions.md index b0d81274..7a45dad8 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/05-write-actions.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/skills/04-write-actions.md @@ -1,11 +1,11 @@ -# 05 - Write action files +# 04 - Write action files One file per action in the plan, written under each confirmed tool's skills root. ## Inputs -- `action_plan` (from 03) -- `files_written`, `blocked_tools` (from 04) +- `action_plan` (from 02) +- `files_written`, `blocked_tools` (from 03) - `target_base` (from 01). Empty string means project root; `plugins/<plugin-name>/` means write under that plugin. Received via conversation context alongside `files_written`. ## Outputs @@ -34,7 +34,7 @@ Action files written under each confirmed tool's skills root. Example layout for ## Process 1. For each confirmed tool from `files_written`, resolve the skill root `<tool skills root>/<skill_name>/` from `@../../references/ai-mapping.md`. Prepend `target_base` to every write path (e.g. when `target_base = ""`: `.claude/skills/<skill_name>/actions/`; when `target_base = "plugins/my-plugin/"`: `plugins/my-plugin/.claude/skills/<skill_name>/actions/`). Paths are CWD-relative; write them directly under the workspace root. Never resolve paths relative to the plugin install directory. Skip any tool in `blocked_tools`. -2. For each action in the plan: copy `@../../assets/skills/action-template.md`, fill each `<placeholder>` per its inline annotation. Transcribe the `test` cell from 03 **verbatim** into the `## Test` section. Write the file to `<target_base><tool skills root>/<skill_name>/actions/<NN>-<slug>.md`. +2. For each action in the plan: copy `@../../assets/skills/action-template.md`, fill each `<placeholder>` per its inline annotation. Transcribe the `test` cell from 02 **verbatim** into the `## Test` section. Write the file to `<target_base><tool skills root>/<skill_name>/actions/<NN>-<slug>.md`. 3. Secrets are **per-skill, never at repo root**. Each skill owns `<skill>/.env` (gitignored, real keys, one `KEY=value` per line) and `<skill>/.env.local` (gitignored, for each key: generation URL + one-line how-to with scopes / plan tier / dashboard path). Add the `<target_base><skill>/.env` and `<target_base><skill>/.env.local` patterns to `.gitignore`. Non-secret data follows R7 (see `@../../references/skill-authoring.md`): cross-skill -> shared root folder; skill-specific -> `<target_base><skill>/assets/` or `<target_base><skill>/references/`. Under Model Y (multi-tool fan-out), secrets fan out with the rest of the skill content: one `.env` per confirmed tool's skill root (e.g. `<target_base>.claude/skills/<name>/.env`, `<target_base>.cursor/skills/<name>/.env`, `<target_base>.github/skills/<name>/.env`). Add a `.gitignore` pattern for each location to `.gitignore`. The user is responsible for populating and protecting each file at its target location; this action only creates the file and records the `.gitignore` entry. diff --git a/plugins/aidd-context/skills/03-context-generate/actions/skills/06-validate.md b/plugins/aidd-context/skills/03-context-generate/actions/skills/05-validate.md similarity index 84% rename from plugins/aidd-context/skills/03-context-generate/actions/skills/06-validate.md rename to plugins/aidd-context/skills/03-context-generate/actions/skills/05-validate.md index c61a6bd0..2be7df51 100644 --- a/plugins/aidd-context/skills/03-context-generate/actions/skills/06-validate.md +++ b/plugins/aidd-context/skills/03-context-generate/actions/skills/05-validate.md @@ -1,4 +1,4 @@ -# 06 - Validate end-to-end +# 05 - Validate end-to-end Spawn one agent per action of the target skill, run its `## Test`, aggregate into a single pass/fail table. @@ -16,7 +16,7 @@ A single markdown table delivered to the user. One row per action, in numeric or | `fetch-week-tasks` | Queried Notion DB for user X, expected ≥ 1 result | ✅ | | `archive-channel` | - | ⏭️ | -`✅` pass, `❌` fail, `⏭️` skipped (e.g. action 02 when manual-only). +`✅` pass, `❌` fail, `⏭️` skipped (e.g. an action that self-skips for the current inputs). ## Process @@ -29,8 +29,7 @@ A single markdown table delivered to the user. One row per action, in numeric or - **Instruction** : execute `## Process`, then run the `## Test`; report pass/fail and the cause if fail in ≤ 100 words. 2. Capture for each action: slug, the `## Test` sentence verbatim, status. Add a row. 3. On `❌`, the agent diagnoses the root cause, **resolves it in the real environment** (install MCP, generate API key per `.env.local`, authenticate, etc.), then **patches the action source file on disk** to point to the working solution. Re-run `## Test` via a fresh agent. Repeat until ✅. The patched action is the documentation - no separate change log. -4. If `disable-model-invocation: false` in `<skill_path>/SKILL.md`: for each scenario in `<skill_path>/evals/scenarios.json`, spawn one fresh agent that reproduces the prompt verbatim and reports whether the dispatched action matches `expect_action`. Add one row per scenario. -5. Always deliver the table at the end, even if every row is ✅. +4. Always deliver the table at the end, even if every row is ✅. ## Test diff --git a/plugins/aidd-context/skills/03-context-generate/assets/plugins/plugin-readme-template.md b/plugins/aidd-context/skills/03-context-generate/assets/plugins/plugin-readme-template.md index 8bba5ea6..8bfd047c 100644 --- a/plugins/aidd-context/skills/03-context-generate/assets/plugins/plugin-readme-template.md +++ b/plugins/aidd-context/skills/03-context-generate/assets/plugins/plugin-readme-template.md @@ -15,6 +15,6 @@ claude plugin install {{plugin_name}} ## Layout - `.claude-plugin/plugin.json` - plugin manifest. -- `skills/` - one folder per skill (each with `SKILL.md`, `actions/`, optional `assets/`, `references/`, `evals/`). +- `skills/` - one folder per skill (each with `SKILL.md`, `actions/`, optional `assets/`, `references/`). - `agents/` - flat agent files (optional). - `hooks/` - hook scripts and `hooks.json` (optional). diff --git a/plugins/aidd-context/skills/03-context-generate/assets/skills/evals-template.md b/plugins/aidd-context/skills/03-context-generate/assets/skills/evals-template.md deleted file mode 100644 index 8af53f62..00000000 --- a/plugins/aidd-context/skills/03-context-generate/assets/skills/evals-template.md +++ /dev/null @@ -1,31 +0,0 @@ -# Evaluation scenarios template - -`evals/scenarios.json` is a minimal probe of trigger correctness. Auto-trigger skills only. - -## Schema - -```json -[ - { "prompt": "<user prompt verbatim>", "expect_action": "<slug>" }, - { "prompt": "<user prompt verbatim>", "expect_action": null } -] -``` - -- `prompt` - string, what a user would actually type. -- `expect_action` - slug of the action to dispatch, or `null` if the skill must NOT trigger. - -## Rules - -- ≥ 3 entries. -- No duplicate prompts. -- Every non-null `expect_action` must match a real action slug. - -## Example (hypothetical `slack` skill) - -```json -[ - { "prompt": "Post this update to #engineering", "expect_action": "post-message" }, - { "prompt": "Get the last 20 messages from #product", "expect_action": "get-history" }, - { "prompt": "Send an email to the team", "expect_action": null } -] -``` diff --git a/plugins/aidd-context/skills/03-context-generate/evals/scenarios.json b/plugins/aidd-context/skills/03-context-generate/evals/scenarios.json deleted file mode 100644 index d6ed044c..00000000 --- a/plugins/aidd-context/skills/03-context-generate/evals/scenarios.json +++ /dev/null @@ -1,20 +0,0 @@ -[ - { "prompt": "Create a new skill for posting to Slack", "expect_action": "capture-intent" }, - { "prompt": "I want to refactor the review skill to add a security action", "expect_action": "capture-intent" }, - { "prompt": "Convert my /commit-message slash command into a proper skill", "expect_action": "capture-intent" }, - { "prompt": "Generate a new skill with SKILL.md and actions", "expect_action": "capture-intent" }, - { "prompt": "Build a new agent file for code review", "expect_action": "capture-intent" }, - { "prompt": "Add an action to an existing skill", "expect_action": "capture-intent" }, - { "prompt": "Generate a new rule file for our project", "expect_action": "capture-intent" }, - { "prompt": "Migrate slash command to a skill", "expect_action": "capture-intent" }, - { "prompt": "Crée un nouveau skill pour notre besoin Slack", "expect_action": "capture-intent" }, - { "prompt": "Génère un agent dédié à la review", "expect_action": "capture-intent" }, - { "prompt": "Refactore ce skill pour ajouter une action", "expect_action": "capture-intent" }, - { "prompt": "Crée un nouvel agent et son skill associé", "expect_action": "capture-intent" }, - { "prompt": "Edit the process step of actions/post-message.md in the slack skill", "expect_action": null }, - { "prompt": "Write an MCP server for my internal CRM", "expect_action": null }, - { "prompt": "Update the project CLAUDE.md with our new doctrine", "expect_action": null }, - { "prompt": "Generate a new agent for this project - I have both .claude/ and .cursor/ directories", "expect_action": "capture-intent" }, - { "prompt": "Create a rule file - this is a fresh repo with no AI tool installed yet", "expect_action": "capture-intent" }, - { "prompt": "Scaffold a plugin for my OpenCode project", "expect_action": "capture-intent" } -] diff --git a/plugins/aidd-context/skills/03-context-generate/references/skill-authoring.md b/plugins/aidd-context/skills/03-context-generate/references/skill-authoring.md index 3fe1d6b9..b62f1c17 100644 --- a/plugins/aidd-context/skills/03-context-generate/references/skill-authoring.md +++ b/plugins/aidd-context/skills/03-context-generate/references/skill-authoring.md @@ -4,7 +4,7 @@ Conventions for every skill the `skills` sub-flow produces. These govern the CLI ## Rules -These rules apply EXCLUSIVELY to generated skills. They do NOT apply to agents, rules, commands, hooks, plugins, or marketplaces - each of those artifact types has its own conventions in `assets/` and `references/`. The `skills` sub-flow MUST enforce R1-R10 on every skill it generates. +These rules apply EXCLUSIVELY to generated skills. They do NOT apply to agents, rules, commands, hooks, plugins, or marketplaces - each of those artifact types has its own conventions in `assets/` and `references/`. The `skills` sub-flow MUST enforce R1-R9 on every skill it generates. - **R1** - SKILL.md is a pure router: description + action table + transversal rules. Zero business logic. - **R2** - One skill = one domain (tool OR activity). Tool -> singular noun (`slack`); activity -> action verb (`review`). See `## Naming` below. @@ -14,8 +14,7 @@ These rules apply EXCLUSIVELY to generated skills. They do NOT apply to agents, - **R6** - Zero duplication. Templates live in `assets/`; actions point to them via `@<path>`. - **R7** - `references/` = documents to READ (conventions, cheatsheets). `assets/` = files to COPY or INJECT (templates, ID tables). - **R8** - Every action has a `## Test`: one sentence describing how to verify its intent - a command to run, a concrete check on the produced artifact, or an observable side-effect (API/MCP/state). -- **R9** - Auto-trigger skills (`disable-model-invocation: false`, default) ship `evals/scenarios.json` = JSON array of at least 3 `{prompt, expect_action}`. Manual-only skills skip. -- **R10** - Generated skills are written in **English only** (frontmatter, body, actions, references, assets). Holds regardless of conversation language. +- **R9** - Generated skills are written in **English only** (frontmatter, body, actions, references, assets). Holds regardless of conversation language. ## Folder layout @@ -26,7 +25,6 @@ These rules apply EXCLUSIVELY to generated skills. They do NOT apply to agents, assets/ (optional) - templates and other files actions COPY / INJECT via @<path> references/ (optional) - documents actions READ when they need the knowledge scripts/ (optional) - skill-local executables called from actions - evals/ (optional) - auto-trigger probes; required when auto-invocation is on README.md (optional) - human-facing summary; never loaded into the model context ``` @@ -85,20 +83,6 @@ Skill-local executables invoked from action `## Process` steps. Never inline `cu Skill secrets (`.env`, `.env.local`) live inside the skill folder, never at repo root, always gitignored. -## `evals/scenarios.json` - -Required for auto-trigger skills (`disable-model-invocation: false`, the default). Minimum 3 entries: - -```json -[ - { "prompt": "<realistic user message>", "expect_action": "<slug or null>" } -] -``` - -`expect_action: null` means the skill must NOT trigger for that prompt. Use both positive and negative cases. - -Manual-only skills (`disable-model-invocation: true`) skip evals. - ## Naming ### Skill names @@ -115,7 +99,7 @@ Two valid patterns, one per domain type. - Numbered prefix (`01-`, `02-`) when either applies: - Execution order is strict (sequential flow), OR - Visual grouping by family improves readability at a glance (e.g. setup -> run -> verify -> cleanup). - In both cases, the slug used everywhere else (SKILL.md action table, `evals/scenarios.json` `expect_action`) is the name without the prefix - validators strip `^\d+-` automatically. + In both cases, the slug used everywhere else (SKILL.md action table) is the name without the prefix - validators strip `^\d+-` automatically. - Avoid vague names: `helper`, `utils`, `main`, `do-stuff`. ### Action table numbering in SKILL.md diff --git a/plugins/aidd-context/skills/04-mermaid/evals/scenarios.json b/plugins/aidd-context/skills/04-mermaid/evals/scenarios.json deleted file mode 100644 index 9e63027e..00000000 --- a/plugins/aidd-context/skills/04-mermaid/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Generate a Mermaid diagram for the authentication flow", "expect_action": "mermaid" }, - { "prompt": "Create a sequence diagram showing the checkout process", "expect_action": "mermaid" }, - { "prompt": "Draw a state machine diagram for the order lifecycle", "expect_action": "mermaid" }, - { "prompt": "Make a flowchart of the onboarding steps", "expect_action": "mermaid" }, - { "prompt": "Visualize the CI/CD pipeline as a Mermaid diagram", "expect_action": "mermaid" }, - { "prompt": "Draw the architecture as a Mermaid diagram", "expect_action": "mermaid" }, - { "prompt": "Produce a class diagram in Mermaid syntax", "expect_action": "mermaid" }, - { "prompt": "Generate ER diagram in Mermaid for the users schema", "expect_action": "mermaid" }, - { "prompt": "Fais un diagramme Mermaid de l'auth flow", "expect_action": "mermaid" }, - { "prompt": "Crée un diagramme de séquence Mermaid", "expect_action": "mermaid" }, - { "prompt": "Génère un diagramme d'état Mermaid pour la commande", "expect_action": "mermaid" }, - { "prompt": "Visualise le pipeline en diagramme Mermaid", "expect_action": "mermaid" }, - { "prompt": "Update memory bank with the new architecture decision", "expect_action": null }, - { "prompt": "Generate a plan for the notifications feature", "expect_action": null }, - { "prompt": "List all installed skills", "expect_action": null }, - { "prompt": "Commit my changes", "expect_action": null } -] - diff --git a/plugins/aidd-context/skills/05-learn/evals/scenarios.json b/plugins/aidd-context/skills/05-learn/evals/scenarios.json deleted file mode 100644 index 2cd8a2a5..00000000 --- a/plugins/aidd-context/skills/05-learn/evals/scenarios.json +++ /dev/null @@ -1,21 +0,0 @@ -[ - { "prompt": "Update the memory bank after implementing the auth feature", "expect_action": "scope" }, - { "prompt": "Capture what we learned from this sprint into project memory", "expect_action": "scope" }, - { "prompt": "Store the decision to use Redis for caching as an ADR", "expect_action": "scope" }, - { "prompt": "Update the coding rules with the new pattern we adopted", "expect_action": "scope" }, - { "prompt": "Learn from this implementation and update project context", "expect_action": "scope" }, - { "prompt": "For next we have to keep this rule: always check context-generate when creating skills, agents, commands, or rules", "expect_action": "scope" }, - { "prompt": "From now on, all DB writes go through the repository layer, never the ORM directly", "expect_action": "scope" }, - { "prompt": "We switched from REST to GraphQL because of nested query needs", "expect_action": "scope" }, - { "prompt": "Next time we should mock the network layer at the fetch boundary, not at the component", "expect_action": "scope" }, - { "prompt": "Deprecated the legacy auth middleware, all routes now use the new guard", "expect_action": "scope" }, - { "prompt": "Remember that I prefer concise answers", "expect_action": null }, - { "prompt": "Always call me Baptiste, never Bap", "expect_action": null }, - { "prompt": "Fixed a typo in README", "expect_action": null }, - { "prompt": "Renamed variable foo to bar", "expect_action": null }, - { "prompt": "Bumped lodash to 4.17.21", "expect_action": null }, - { "prompt": "Refactored auth middleware", "expect_action": null }, - { "prompt": "Generate a Mermaid diagram for the auth flow", "expect_action": null }, - { "prompt": "Help me discover what skills are available", "expect_action": null }, - { "prompt": "Create an implementation plan for the dashboard", "expect_action": null } -] diff --git a/plugins/aidd-context/skills/06-discovery/evals/scenarios.json b/plugins/aidd-context/skills/06-discovery/evals/scenarios.json deleted file mode 100644 index 7feee35c..00000000 --- a/plugins/aidd-context/skills/06-discovery/evals/scenarios.json +++ /dev/null @@ -1,37 +0,0 @@ -[ - { "prompt": "What skills are available for me to use?", "expect_action": "find-skill" }, - { "prompt": "Help me find the right skill for writing tests", "expect_action": "find-skill" }, - { "prompt": "Which skill should I use to create a plan?", "expect_action": "find-skill" }, - { "prompt": "List all installed skills and what they do", "expect_action": "find-skill" }, - { "prompt": "Show me the skills", "expect_action": "find-skill" }, - { "prompt": "Which agent handles pull request reviews?", "expect_action": "find-agent" }, - { "prompt": "List installed agents", "expect_action": "find-agent" }, - { "prompt": "Enumerate agents", "expect_action": "find-agent" }, - { "prompt": "What slash commands can I run?", "expect_action": "find-command" }, - { "prompt": "Which command opens a pull request?", "expect_action": "find-command" }, - { "prompt": "Show me the commands", "expect_action": "find-command" }, - { "prompt": "What plugins are installed?", "expect_action": "find-plugin" }, - { "prompt": "Which plugin handles VCS operations?", "expect_action": "find-plugin" }, - { "prompt": "List plugins", "expect_action": "find-plugin" }, - { "prompt": "What MCP servers are connected?", "expect_action": "find-mcp" }, - { "prompt": "Which external system can I use to search Jira issues?", "expect_action": "find-mcp" }, - { "prompt": "List the MCPs", "expect_action": "find-mcp" }, - { "prompt": "What rules apply to TypeScript API files?", "expect_action": "find-rule" }, - { "prompt": "List the project rules", "expect_action": "find-rule" }, - { "prompt": "Show me the rules", "expect_action": "find-rule" }, - { "prompt": "Which Cursor rule covers Mermaid diagrams?", "expect_action": "find-rule" }, - { "prompt": "What hooks are installed?", "expect_action": "find-hook" }, - { "prompt": "List hooks", "expect_action": "find-hook" }, - { "prompt": "Show me the hooks", "expect_action": "find-hook" }, - { "prompt": "Which hook fires on UserPromptSubmit?", "expect_action": "find-hook" }, - { "prompt": "Enumerate session hooks", "expect_action": "find-hook" }, - { "prompt": "What memory files do we have?", "expect_action": "find-memory" }, - { "prompt": "Which memory file describes the architecture?", "expect_action": "find-memory" }, - { "prompt": "List the project memory bank", "expect_action": "find-memory" }, - { "prompt": "Show me the memory files", "expect_action": "find-memory" }, - { "prompt": "Update the memory bank with the new ADR", "expect_action": null }, - { "prompt": "Generate a Mermaid diagram for the auth flow", "expect_action": null }, - { "prompt": "Write an implementation plan for the checkout feature", "expect_action": null }, - { "prompt": "Explain this code", "expect_action": null }, - { "prompt": "Explique-moi le code", "expect_action": null } -] diff --git a/plugins/aidd-dev/CATALOG.md b/plugins/aidd-dev/CATALOG.md index bb39669d..bcbbc04f 100644 --- a/plugins/aidd-dev/CATALOG.md +++ b/plugins/aidd-dev/CATALOG.md @@ -47,7 +47,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [03-implement.md](skills/00-sdlc/actions/03-implement.md) | - | | `actions` | [04-review.md](skills/00-sdlc/actions/04-review.md) | - | | `actions` | [05-ship.md](skills/00-sdlc/actions/05-ship.md) | - | -| `evals` | [scenarios.json](skills/00-sdlc/evals/scenarios.json) | - | | `-` | [README.md](skills/00-sdlc/README.md) | - | | `-` | [SKILL.md](skills/00-sdlc/SKILL.md) | `Pure orchestrator for the full AIDD development flow. Use when a human (or Gardener) needs to take a free-form request from idea to shipped code, end-to-end. Coordinates spec generation, planning, implementation, review, and shipping by composing other skills and agents. Supports two modes - `auto` (default, no human interaction) and `interactive` (pauses for human confirmation at key gates). Holds no business logic of its own; every step is delegated.` | @@ -61,7 +60,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `assets` | [master-plan-template.md](skills/01-plan/assets/master-plan-template.md) | `Parent plan template orchestrating multiple child plans with validation gates` | - | | `assets` | [plan-template.md](skills/01-plan/assets/plan-template.md) | `Living implementation plan - frozen objective, phases, and append-only execution Log. Used as input artifact AND as the autonomous-loop tracking file.` | - | | `assets` | [tech-choice-template.md](skills/01-plan/assets/tech-choice-template.md) | `Technology selection and comparison template` | - | -| `evals` | [scenarios.json](skills/01-plan/evals/scenarios.json) | - | - | | `-` | [README.md](skills/01-plan/README.md) | - | - | | `references` | [mermaid-conventions.md](skills/01-plan/references/mermaid-conventions.md) | `Rules for generating valid, high-quality Mermaid diagrams. Apply when creating or reviewing any Mermaid diagram (flowchart, state, ER, sequence, gantt).` | - | | `-` | [SKILL.md](skills/01-plan/SKILL.md) | `Generate technical implementation plans, define component behaviors, and extract design details from images.` | - | @@ -71,7 +69,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| | `actions` | [01-implement.md](skills/02-implement/actions/01-implement.md) | - | -| `evals` | [scenarios.json](skills/02-implement/evals/scenarios.json) | - | | `-` | [README.md](skills/02-implement/README.md) | - | | `-` | [SKILL.md](skills/02-implement/SKILL.md) | `Execute an implementation plan phase by phase via the implementer agent, iterating until 100% completeness.` | @@ -83,7 +80,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-assert-architecture.md](skills/03-assert/actions/02-assert-architecture.md) | - | | `actions` | [03-assert-frontend.md](skills/03-assert/actions/03-assert-frontend.md) | - | | `assets` | [task-template.md](skills/03-assert/assets/task-template.md) | `Task tracking system to ensure all tasks are categorized and addressed` | -| `evals` | [scenarios.json](skills/03-assert/evals/scenarios.json) | - | | `-` | [README.md](skills/03-assert/README.md) | - | | `-` | [SKILL.md](skills/03-assert/SKILL.md) | `Assert features work as intended - general assertions, architecture conformance, and frontend UI validation.` | @@ -99,7 +95,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [06-tests.md](skills/04-audit/actions/06-tests.md) | - | - | | `actions` | [07-ui.md](skills/04-audit/actions/07-ui.md) | - | - | | `assets` | [audit-template.md](skills/04-audit/assets/audit-template.md) | `Codebase audit report template` | - | -| `evals` | [scenarios.json](skills/04-audit/evals/scenarios.json) | - | - | | `-` | [README.md](skills/04-audit/README.md) | - | - | | `-` | [SKILL.md](skills/04-audit/SKILL.md) | `Read-only codebase audit across quality pillars (code-quality, architecture, security, dependencies, performance, tests, ui). Diagnoses and reports findings; never edits code. Use when the user wants to assess, audit, or health-check a codebase or one dimension of it, then hands off to the act-skills (refactor, test, impeccable) to fix. Do NOT use for fixing the findings (hand off to refactor/test/impeccable), per-PR code review (use 05-review), or validating that a feature works (use 03-assert).` | - | @@ -111,7 +106,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-review-functional.md](skills/05-review/actions/02-review-functional.md) | - | - | | `assets` | [review-code-template.md](skills/05-review/assets/review-code-template.md) | `Code review report template for a diff` | - | | `assets` | [review-functional-template.md](skills/05-review/assets/review-functional-template.md) | `Functional review report template for a diff against a plan` | - | -| `evals` | [scenarios.json](skills/05-review/evals/scenarios.json) | - | - | | `-` | [README.md](skills/05-review/README.md) | - | - | | `-` | [SKILL.md](skills/05-review/SKILL.md) | `Read-only review of a diff (a PR or working changes) - code quality against project rules, and feature behavior against the plan's acceptance criteria. Surfaces findings with a verdict; never patches. Use to review changes in progress. Do NOT use for a whole-codebase health check (use 04-audit), fixing the findings (hand off to 07-refactor / 02-implement / 08-debug), or validating a feature runs (use 03-assert).` | - | @@ -121,7 +115,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---| | `actions` | [01-test.md](skills/06-test/actions/01-test.md) | - | | `actions` | [02-test-journey.md](skills/06-test/actions/02-test-journey.md) | - | -| `evals` | [scenarios.json](skills/06-test/evals/scenarios.json) | - | | `-` | [README.md](skills/06-test/README.md) | - | | `-` | [SKILL.md](skills/06-test/SKILL.md) | `Write and iterate on tests until they pass, and validate user journeys end-to-end in the browser.` | @@ -133,7 +126,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-security.md](skills/07-refactor/actions/02-security.md) | - | | `actions` | [03-cleanup.md](skills/07-refactor/actions/03-cleanup.md) | - | | `actions` | [04-architecture.md](skills/07-refactor/actions/04-architecture.md) | - | -| `evals` | [scenarios.json](skills/07-refactor/evals/scenarios.json) | - | | `-` | [README.md](skills/07-refactor/README.md) | - | | `-` | [SKILL.md](skills/07-refactor/SKILL.md) | `Improve code without breaking behavior across four axes - cleanup (clean-code + tech debt), performance, security, architecture. Scans and fixes, or fixes the findings of an audit report pushed in by the caller. Use when the user wants to refactor, clean up, optimize, harden, or restructure code. Do NOT use for read-only diagnosis (use 04-audit), adding tests (use 06-test), or UI redesign (use the impeccable skill).` | @@ -145,7 +137,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-debug.md](skills/08-debug/actions/02-debug.md) | - | | `actions` | [03-reflect-issue.md](skills/08-debug/actions/03-reflect-issue.md) | - | | `assets` | [task-template.md](skills/08-debug/assets/task-template.md) | `Task tracking system to ensure all tasks are categorized and addressed` | -| `evals` | [scenarios.json](skills/08-debug/evals/scenarios.json) | - | | `-` | [README.md](skills/08-debug/README.md) | - | | `references` | [mermaid-conventions.md](skills/08-debug/references/mermaid-conventions.md) | `Rules for generating valid, high-quality Mermaid diagrams. Apply when creating or reviewing any Mermaid diagram (flowchart, state, ER, sequence, gantt).` | | `-` | [SKILL.md](skills/08-debug/SKILL.md) | `Reproduce and fix bugs systematically using test-driven workflow, root cause analysis, and hypothesis validation.` | @@ -158,7 +149,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-auto-accept.md](skills/09-for-sure/actions/02-auto-accept.md) | - | | `actions` | [03-autonomous-loop.md](skills/09-for-sure/actions/03-autonomous-loop.md) | - | | `assets` | [plan-template.md](skills/09-for-sure/assets/plan-template.md) | - | -| `evals` | [scenarios.json](skills/09-for-sure/evals/scenarios.json) | - | | `-` | [README.md](skills/09-for-sure/README.md) | - | | `-` | [SKILL.md](skills/09-for-sure/SKILL.md) | `Iterative agent loop that tracks attempts and retries until a success condition is met. Use when the user says "for sure", "make sure", "keep trying until", "loop until done", "don't stop until", or needs guaranteed completion of a task with explicit success criteria.` | diff --git a/plugins/aidd-dev/skills/00-sdlc/evals/scenarios.json b/plugins/aidd-dev/skills/00-sdlc/evals/scenarios.json deleted file mode 100644 index f52252ee..00000000 --- a/plugins/aidd-dev/skills/00-sdlc/evals/scenarios.json +++ /dev/null @@ -1,15 +0,0 @@ -[ - { "prompt": "/sdlc add a /health endpoint", "expect_action": "spec" }, - { "prompt": "Run the full dev flow on this request", "expect_action": "spec" }, - { "prompt": "Ship this feature from idea to PR", "expect_action": "spec" }, - { "prompt": "/sdlc interactive add a /health endpoint", "expect_action": "spec" }, - { "prompt": "Run the SDLC in interactive mode on this request", "expect_action": "spec" }, - { "prompt": "Interactive sdlc, pause at every gate", "expect_action": "spec" }, - { "prompt": "Plan this ticket end to end", "expect_action": "plan" }, - { "prompt": "Resume the SDLC run from this plan", "expect_action": "implement" }, - { "prompt": "Review the milestone output", "expect_action": "review" }, - { "prompt": "Open the PR for this branch", "expect_action": "ship" }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Draft a PRD for billing", "expect_action": null }, - { "prompt": "Implement milestone M3 in isolation", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/01-plan/evals/scenarios.json b/plugins/aidd-dev/skills/01-plan/evals/scenarios.json deleted file mode 100644 index 6354a25b..00000000 --- a/plugins/aidd-dev/skills/01-plan/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Generate a technical plan for the authentication feature", "expect_action": "plan" }, - { "prompt": "Create an implementation plan from the spec", "expect_action": "plan" }, - { "prompt": "Plan the refactoring of the payment module", "expect_action": "plan" }, - { "prompt": "Define component behavior for the checkout flow", "expect_action": "components-behavior" }, - { "prompt": "Create a state machine for the onboarding wizard", "expect_action": "components-behavior" }, - { "prompt": "Extract component details from this design screenshot", "expect_action": "image-extract-details" }, - { "prompt": "Analyze this Figma image and list all components", "expect_action": "image-extract-details" }, - { "prompt": "Rédige un plan technique pour l'auth", "expect_action": "plan" }, - { "prompt": "Génère le plan d'implémentation depuis la spec", "expect_action": "plan" }, - { "prompt": "Crée le plan d'implémentation phase par phase", "expect_action": "plan" }, - { "prompt": "Définis le comportement du composant checkout", "expect_action": "components-behavior" }, - { "prompt": "Extrais les composants de cette image Figma", "expect_action": "image-extract-details" }, - { "prompt": "Write the code for the login page", "expect_action": null }, - { "prompt": "Review my pull request", "expect_action": null }, - { "prompt": "Debug the failing test in auth.test.ts", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/02-implement/evals/scenarios.json b/plugins/aidd-dev/skills/02-implement/evals/scenarios.json deleted file mode 100644 index 09a198da..00000000 --- a/plugins/aidd-dev/skills/02-implement/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Implement the plan for the payment feature", "expect_action": "implement" }, - { "prompt": "Execute this implementation plan phase by phase", "expect_action": "implement" }, - { "prompt": "Run the plan end-to-end", "expect_action": "implement" }, - { "prompt": "Implement phase 2 of the auth plan", "expect_action": "implement" }, - { "prompt": "Start implementing from the plan file in aidd_docs/plans/", "expect_action": "implement" }, - { "prompt": "Execute the implementation plan phase by phase", "expect_action": "implement" }, - { "prompt": "Implement from the plan in aidd_docs/plans", "expect_action": "implement" }, - { "prompt": "Run the implementation plan end-to-end", "expect_action": "implement" }, - { "prompt": "Implement the plan via the implementer agent", "expect_action": "implement" }, - { "prompt": "Exécute le plan d'implémentation phase par phase", "expect_action": "implement" }, - { "prompt": "Lance l'implémentation depuis le plan", "expect_action": "implement" }, - { "prompt": "Implémente le plan via l'agent implementer", "expect_action": "implement" }, - { "prompt": "Create a plan for the notifications feature", "expect_action": null }, - { "prompt": "Review the code I just wrote", "expect_action": null }, - { "prompt": "Fix the bug in the payment service", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/03-assert/evals/scenarios.json b/plugins/aidd-dev/skills/03-assert/evals/scenarios.json deleted file mode 100644 index 286f4815..00000000 --- a/plugins/aidd-dev/skills/03-assert/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Assert the login feature works as intended", "expect_action": "assert" }, - { "prompt": "Verify the payment flow is correctly implemented", "expect_action": "assert" }, - { "prompt": "Check that this feature matches the acceptance criteria", "expect_action": "assert" }, - { "prompt": "Assert architecture conformance for the new module", "expect_action": "assert-architecture" }, - { "prompt": "Check that the code follows the layered architecture rules", "expect_action": "assert-architecture" }, - { "prompt": "Validate the frontend UI against the design", "expect_action": "assert-frontend" }, - { "prompt": "Assert the checkout page looks correct in the browser", "expect_action": "assert-frontend" }, - { "prompt": "Vérifie que la feature login fonctionne comme prévu", "expect_action": "assert" }, - { "prompt": "Assert conformité architecturale du module", "expect_action": "assert-architecture" }, - { "prompt": "Valide la conformité de l'UI au design", "expect_action": "assert-frontend" }, - { "prompt": "Assert acceptance criteria for the auth feature", "expect_action": "assert" }, - { "prompt": "Verify feature works against intended behavior", "expect_action": "assert" }, - { "prompt": "Write tests for the login feature", "expect_action": null }, - { "prompt": "Review my code for quality issues", "expect_action": null }, - { "prompt": "Generate a plan for the dashboard", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/04-audit/evals/scenarios.json b/plugins/aidd-dev/skills/04-audit/evals/scenarios.json deleted file mode 100644 index e60ea226..00000000 --- a/plugins/aidd-dev/skills/04-audit/evals/scenarios.json +++ /dev/null @@ -1,21 +0,0 @@ -[ - { "prompt": "Audit code quality and tech debt in the auth module", "expect_action": "code-quality" }, - { "prompt": "Find dead code, duplication and complexity hotspots", "expect_action": "code-quality" }, - { "prompt": "Trouve le code mort et la duplication", "expect_action": "code-quality" }, - { "prompt": "Audit the architecture conformance against our ADRs", "expect_action": "architecture" }, - { "prompt": "Check module coupling and layering violations", "expect_action": "architecture" }, - { "prompt": "Security audit of the API endpoints", "expect_action": "security" }, - { "prompt": "Audit for OWASP risks, authz gaps and committed secrets", "expect_action": "security" }, - { "prompt": "Audit de securite du projet", "expect_action": "security" }, - { "prompt": "Audit dependencies for known CVEs and outdated packages", "expect_action": "dependencies" }, - { "prompt": "Check vulnerable and unused dependencies", "expect_action": "dependencies" }, - { "prompt": "Performance audit: find N+1 queries and hot paths", "expect_action": "performance" }, - { "prompt": "Audit bundle size and slow render paths", "expect_action": "performance" }, - { "prompt": "Audit test coverage on critical paths", "expect_action": "tests" }, - { "prompt": "Where are tests flaky or missing", "expect_action": "tests" }, - { "prompt": "Audit the UI for accessibility and missing loading states", "expect_action": "ui" }, - { "prompt": "Check design-system drift and responsive gaps", "expect_action": "ui" }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Fix this specific bug in checkout", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/05-review/evals/scenarios.json b/plugins/aidd-dev/skills/05-review/evals/scenarios.json deleted file mode 100644 index 260793d8..00000000 --- a/plugins/aidd-dev/skills/05-review/evals/scenarios.json +++ /dev/null @@ -1,16 +0,0 @@ -[ - { "prompt": "Review the code I just implemented", "expect_action": "review-code" }, - { "prompt": "Do a code review of the PR changes", "expect_action": "review-code" }, - { "prompt": "Check code quality against project rules", "expect_action": "review-code" }, - { "prompt": "Review whether the feature behavior matches the plan", "expect_action": "review-functional" }, - { "prompt": "Validate the implementation against the acceptance criteria in the plan", "expect_action": "review-functional" }, - { "prompt": "Functional review of the checkout feature", "expect_action": "review-functional" }, - { "prompt": "Code review against project conventions", "expect_action": "review-code" }, - { "prompt": "Functional review against the plan specification", "expect_action": "review-functional" }, - { "prompt": "Revue de code contre les règles du projet", "expect_action": "review-code" }, - { "prompt": "Revue fonctionnelle contre le plan d'implémentation", "expect_action": "review-functional" }, - { "prompt": "Vérifie la qualité du code selon nos rules", "expect_action": "review-code" }, - { "prompt": "Generate an implementation plan for the dashboard", "expect_action": null }, - { "prompt": "Write unit tests for the auth module", "expect_action": null }, - { "prompt": "Fix the bug in the payment service", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/06-test/evals/scenarios.json b/plugins/aidd-dev/skills/06-test/evals/scenarios.json deleted file mode 100644 index cc9bdba3..00000000 --- a/plugins/aidd-dev/skills/06-test/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Write tests for the authentication module", "expect_action": "test" }, - { "prompt": "Add unit tests for the payment service", "expect_action": "test" }, - { "prompt": "Iterate on tests until they pass", "expect_action": "test" }, - { "prompt": "Write and run tests for this feature", "expect_action": "test" }, - { "prompt": "Validate the user journey for the checkout flow in the browser", "expect_action": "test-journey" }, - { "prompt": "Test the end-to-end user registration flow", "expect_action": "test-journey" }, - { "prompt": "Write unit tests for the auth module", "expect_action": "test" }, - { "prompt": "Add coverage tests for the payment service", "expect_action": "test" }, - { "prompt": "Test end-to-end user journey via the browser", "expect_action": "test-journey" }, - { "prompt": "List untested behaviors and iterate on test creation", "expect_action": "test" }, - { "prompt": "Écris les tests unitaires du module auth", "expect_action": "test" }, - { "prompt": "Test le parcours user en bout-en-bout dans le browser", "expect_action": "test-journey" }, - { "prompt": "Itère sur les tests jusqu'à ce qu'ils passent", "expect_action": "test" }, - { "prompt": "Valide le user journey du checkout", "expect_action": "test-journey" }, - { "prompt": "Review my code for quality issues", "expect_action": null }, - { "prompt": "Debug the failing login test", "expect_action": null }, - { "prompt": "Optimize the database queries", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/07-refactor/evals/scenarios.json b/plugins/aidd-dev/skills/07-refactor/evals/scenarios.json deleted file mode 100644 index 887dab55..00000000 --- a/plugins/aidd-dev/skills/07-refactor/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Optimize the performance of the product listing query", "expect_action": "performance" }, - { "prompt": "Profile and fix slow database calls in the dashboard", "expect_action": "performance" }, - { "prompt": "Improve performance of the image upload pipeline", "expect_action": "performance" }, - { "prompt": "Optimise la performance du module produit", "expect_action": "performance" }, - { "prompt": "Fix security vulnerabilities in the auth module", "expect_action": "security" }, - { "prompt": "Run an OWASP audit and harden the API endpoints", "expect_action": "security" }, - { "prompt": "Corrige les vulnerabilites OWASP du module auth", "expect_action": "security" }, - { "prompt": "Clean up this module: rename, extract, remove dead code", "expect_action": "cleanup" }, - { "prompt": "Reduce the complexity and duplication in the checkout service", "expect_action": "cleanup" }, - { "prompt": "Refactor this file for readability, kill the magic numbers", "expect_action": "cleanup" }, - { "prompt": "Nettoie le code mort et la duplication du service paiement", "expect_action": "cleanup" }, - { "prompt": "Restructure the layers, the views import the database directly", "expect_action": "architecture" }, - { "prompt": "Decouple this god-module and fix the dependency direction", "expect_action": "architecture" }, - { "prompt": "Enforce the documented architecture boundaries", "expect_action": "architecture" }, - { "prompt": "Write tests for the payment service", "expect_action": null }, - { "prompt": "Review my pull request", "expect_action": null }, - { "prompt": "Debug the failing authentication test", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/08-debug/evals/scenarios.json b/plugins/aidd-dev/skills/08-debug/evals/scenarios.json deleted file mode 100644 index c406a334..00000000 --- a/plugins/aidd-dev/skills/08-debug/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Reproduce the bug reported in issue #42", "expect_action": "reproduce" }, - { "prompt": "Write a failing test that reproduces the login crash", "expect_action": "reproduce" }, - { "prompt": "Debug the payment service error", "expect_action": "debug" }, - { "prompt": "Figure out what is causing the checkout to fail", "expect_action": "debug" }, - { "prompt": "Find and fix the root cause of the auth token expiration issue", "expect_action": "debug" }, - { "prompt": "Reflect on what caused the issue and document learnings", "expect_action": "reflect-issue" }, - { "prompt": "Post-mortem: what went wrong with the deployment bug", "expect_action": "reflect-issue" }, - { "prompt": "Find the root cause of this crash", "expect_action": "debug" }, - { "prompt": "Investigate why the test fails", "expect_action": "debug" }, - { "prompt": "Reproduce the failing checkout in a test", "expect_action": "reproduce" }, - { "prompt": "Trouve la cause racine de ce crash", "expect_action": "debug" }, - { "prompt": "Reproduis le bug login dans un test", "expect_action": "reproduce" }, - { "prompt": "Investigue la cause du test qui échoue", "expect_action": "debug" }, - { "prompt": "Post-mortem du bug deployment", "expect_action": "reflect-issue" }, - { "prompt": "Write unit tests for the auth module", "expect_action": null }, - { "prompt": "Review my code quality", "expect_action": null }, - { "prompt": "Create an implementation plan for the dashboard", "expect_action": null } -] diff --git a/plugins/aidd-dev/skills/09-for-sure/evals/scenarios.json b/plugins/aidd-dev/skills/09-for-sure/evals/scenarios.json deleted file mode 100644 index abc7d486..00000000 --- a/plugins/aidd-dev/skills/09-for-sure/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "For sure make all tests pass", "expect_action": "init-tracking" }, - { "prompt": "Make sure the deployment succeeds", "expect_action": "init-tracking" }, - { "prompt": "Keep trying until the CI pipeline is green", "expect_action": "init-tracking" }, - { "prompt": "Loop until done: get the linter to exit 0", "expect_action": "init-tracking" }, - { "prompt": "Don't stop until all tests pass and coverage is above 80%", "expect_action": "init-tracking" }, - { "prompt": "For sure fix this until tests are green", "expect_action": "init-tracking" }, - { "prompt": "Keep looping until the build succeeds", "expect_action": "init-tracking" }, - { "prompt": "Loop until done: all assertions pass", "expect_action": "init-tracking" }, - { "prompt": "Make sure the lint exits zero, retry until it does", "expect_action": "init-tracking" }, - { "prompt": "Ne lâche pas tant que les tests ne passent pas", "expect_action": "init-tracking" }, - { "prompt": "Boucle jusqu'à ce que le CI soit vert", "expect_action": "init-tracking" }, - { "prompt": "For sure jusqu'à ce que ça marche", "expect_action": "init-tracking" }, - { "prompt": "Write a plan for the authentication feature", "expect_action": null }, - { "prompt": "Review my code for quality issues", "expect_action": null }, - { "prompt": "Run the tests once and report results", "expect_action": null } -] diff --git a/plugins/aidd-orchestrator/CATALOG.md b/plugins/aidd-orchestrator/CATALOG.md index 711cf654..16290f8f 100644 --- a/plugins/aidd-orchestrator/CATALOG.md +++ b/plugins/aidd-orchestrator/CATALOG.md @@ -24,7 +24,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| -| `evals` | [scenarios.json](skills/00-async-dev/evals/scenarios.json) | - | | `-` | [README.md](skills/00-async-dev/README.md) | - | | `references` | [routing.md](skills/00-async-dev/references/routing.md) | - | | `-` | [SKILL.md](skills/00-async-dev/SKILL.md) | `Single entry point for the async-dev pipeline (setup, run, review). Hybrid router decides which sub-flow to execute from $ARGUMENTS keyword (`setup` / `run` / `review`), trigger source (label `to-implement` / `to-review`, comment `@claude /implement` / `/review`), repo state (workflow + config presence, PR linked to issue), or natural-language intent. Use when the user says "set up async dev", "run async dev on issue #N", "address review on PR #N", "/async-dev", "claude on issues", or when triggered by a webhook with the matching labels or comments. Do NOT use for plain status checks on the async pipeline or for SDLC orchestration unrelated to issue/PR automation.` | diff --git a/plugins/aidd-orchestrator/skills/00-async-dev/SKILL.md b/plugins/aidd-orchestrator/skills/00-async-dev/SKILL.md index 781f540c..c4bf21d5 100644 --- a/plugins/aidd-orchestrator/skills/00-async-dev/SKILL.md +++ b/plugins/aidd-orchestrator/skills/00-async-dev/SKILL.md @@ -127,4 +127,4 @@ Setup-only templates copied into the target repo by the setup sub-flow: ## Test -Each sub-flow's final action carries a `## Test` block that exercises the full sub-flow with mocked dependencies. The router itself is tested by the dispatch scenarios in `evals/scenarios.json` (signal → expected sub-flow). +Each sub-flow's final action carries a `## Test` block that exercises the full sub-flow with mocked dependencies. diff --git a/plugins/aidd-orchestrator/skills/00-async-dev/evals/scenarios.json b/plugins/aidd-orchestrator/skills/00-async-dev/evals/scenarios.json deleted file mode 100644 index 91333a0e..00000000 --- a/plugins/aidd-orchestrator/skills/00-async-dev/evals/scenarios.json +++ /dev/null @@ -1,210 +0,0 @@ -[ - { - "prompt": "async dev", - "expect_action": "dispatch" - }, - { - "prompt": "/async-dev", - "expect_action": "dispatch" - }, - { - "prompt": "set up async dev and run on issue 42", - "expect_action": "dispatch" - }, - { - "prompt": "claude on issues", - "expect_action": "dispatch" - }, - { - "prompt": "process the async queue", - "expect_action": "dispatch" - }, - { - "prompt": "iterate on PR 17", - "expect_action": "dispatch" - }, - { - "prompt": "is async dev configured here?", - "expect_action": "dispatch" - }, - { - "prompt": "Setup async dev in this repo", - "expect_action": null - }, - { - "prompt": "Run async pipeline on issue #42", - "expect_action": null - }, - { - "prompt": "Handle review comments on PR 17", - "expect_action": null - }, - { - "prompt": "Setup async dev in this repo", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Configure aidd-orchestrator plugin", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Install async-dev so Claude picks up ready issues automatically", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Generate the GitHub Actions workflow for async dev", - "expect_action": "setup/generate-workflow" - }, - { - "prompt": "Install aidd-orchestrator end-to-end", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Configure async-dev pipeline in this repo", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Setup auto-implementation orchestration on issues", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Wire up async-dev workflow in GitHub Actions", - "expect_action": "setup/generate-workflow" - }, - { - "prompt": "Installe async-dev dans ce repo", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Configure aidd-orchestrator pour ce projet", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Met en place auto-implémentation sur issues", - "expect_action": "setup/detect-context" - }, - { - "prompt": "Génère le workflow GitHub Actions async-dev", - "expect_action": "setup/generate-workflow" - }, - { - "prompt": "Run the async pipeline on issue #42", - "expect_action": "setup/" - }, - { - "prompt": "Handle the review comments on the open PR", - "expect_action": "setup/" - }, - { - "prompt": "Run async dev on issue #42", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Process the async queue now", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Implement the next ready issue with Claude", - "expect_action": "run/poll-ready" - }, - { - "prompt": "@claude /implement on this issue", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Poll issues labeled to-implement", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Process issues with the to-implement label", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Run the async-dev pipeline cycle", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Trigger async dev to process ready issues", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Lance le pipeline async-dev", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Traite les issues to-implement", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Run async dev queue process", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Process to-implement labeled issues", - "expect_action": "run/poll-ready" - }, - { - "prompt": "Setup the aidd-orchestrator plugin", - "expect_action": "run/" - }, - { - "prompt": "Handle the review comments on PR #117", - "expect_action": "run/" - }, - { - "prompt": "Handle review comments on PR #117", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Address review feedback automatically on the open PR", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Iterate fixes on PR 89 until reviewer approves or max retries", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Process review feedback loop on PR 42", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Address reviewer comments on the open async-dev PR", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Loop until reviewer approves the async-dev PR", - "expect_action": "review/collect-comments" - }, - { - "prompt": "@claude /review on PR 42", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Iterate review-fix loop until done", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Adresse les comments de la PR async-dev", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Itère sur les retours reviewer du PR ouvert", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Gère le loop review-fix async-dev", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Process le feedback du reviewer en boucle", - "expect_action": "review/collect-comments" - }, - { - "prompt": "Setup async dev plugin", - "expect_action": "review/" - }, - { - "prompt": "Run the async pipeline on issue #50", - "expect_action": "review/" - } -] diff --git a/plugins/aidd-pm/CATALOG.md b/plugins/aidd-pm/CATALOG.md index 6680201b..e2860668 100644 --- a/plugins/aidd-pm/CATALOG.md +++ b/plugins/aidd-pm/CATALOG.md @@ -28,7 +28,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| | `actions` | [01-ticket-info.md](skills/01-ticket-info/actions/01-ticket-info.md) | - | -| `evals` | [scenarios.json](skills/01-ticket-info/evals/scenarios.json) | - | | `-` | [README.md](skills/01-ticket-info/README.md) | - | | `-` | [SKILL.md](skills/01-ticket-info/SKILL.md) | `Retrieve and display ticket information from the configured ticketing tool. Use when the user says "ticket info", "show ticket", "get ticket", "ticket details", "what's <id>", or invokes `/ticket-info`. Do NOT use for creating issues, commenting on tickets, changing status, or reassigning.` | @@ -38,7 +37,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---|---| | `actions` | [01-create-user-stories.md](skills/02-user-stories-create/actions/01-create-user-stories.md) | - | - | | `assets` | [user-story-template.md](skills/02-user-stories-create/assets/user-story-template.md) | `Template for defining user stories with estimation and acceptance criteria` | - | -| `evals` | [scenarios.json](skills/02-user-stories-create/evals/scenarios.json) | - | - | | `-` | [README.md](skills/02-user-stories-create/README.md) | - | - | | `-` | [SKILL.md](skills/02-user-stories-create/SKILL.md) | `Generate INVEST-compliant user stories from a feature description. Use when the user says "user stories", "create user stories", "write user stories for X", "INVEST stories", "draft stories", or invokes `/user-stories-create`. Do NOT use for writing code, drafting a full PRD, refining a single existing story, or copying ready text into a tracker.` | - | @@ -49,7 +47,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [01-prd.md](skills/03-prd/actions/01-prd.md) | - | - | | `assets` | [prd-template.md](skills/03-prd/assets/prd-template.md) | `Product Requirements Document template (15 sections)` | - | | `assets` | [task-template.md](skills/03-prd/assets/task-template.md) | `Task tracking system to ensure all tasks are categorized and addressed` | - | -| `evals` | [scenarios.json](skills/03-prd/evals/scenarios.json) | - | - | | `-` | [README.md](skills/03-prd/README.md) | - | - | | `-` | [SKILL.md](skills/03-prd/SKILL.md) | `Generate a structured Product Requirements Document from a feature description or user stories, validated with the user before save. Use when the user says "prd", "draft prd", "write prd", "product requirements for X", "generate a prd", or invokes `/prd`. Do NOT use for writing user stories, drafting a technical implementation plan, or writing source code.` | - | @@ -60,7 +57,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [01-build.md](skills/04-spec/actions/01-build.md) | - | | `actions` | [02-refine.md](skills/04-spec/actions/02-refine.md) | - | | `assets` | [spec-template.md](skills/04-spec/assets/spec-template.md) | - | -| `evals` | [scenarios.json](skills/04-spec/evals/scenarios.json) | - | | `-` | [README.md](skills/04-spec/README.md) | - | | `-` | [SKILL.md](skills/04-spec/SKILL.md) | `Generate or refine a project spec from a free-form human request, an existing PRD, or reviewer findings. Use when the user says "draft spec", "spec for X", "refine the spec", "generate spec from prd", "/spec", or when an orchestrator needs a normalized contract before planning. Do NOT use for writing source code, drafting a full PRD, or modifying a validated and locked spec.` | diff --git a/plugins/aidd-pm/skills/01-ticket-info/evals/scenarios.json b/plugins/aidd-pm/skills/01-ticket-info/evals/scenarios.json deleted file mode 100644 index 074b2594..00000000 --- a/plugins/aidd-pm/skills/01-ticket-info/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Show ticket info for ABC-123", "expect_action": "ticket-info" }, - { "prompt": "/ticket-info", "expect_action": "ticket-info" }, - { "prompt": "Get ticket details for the current branch", "expect_action": "ticket-info" }, - { "prompt": "What's ENG-42 about?", "expect_action": "ticket-info" }, - { "prompt": "Show me details of JIRA ticket PROJ-101", "expect_action": "ticket-info" }, - { "prompt": "What does FOO-77 say in the tracker", "expect_action": "ticket-info" }, - { "prompt": "Fetch info for ticket BUG-456", "expect_action": "ticket-info" }, - { "prompt": "Display ticket fields for the active issue", "expect_action": "ticket-info" }, - { "prompt": "Affiche les infos du ticket ABC-99", "expect_action": "ticket-info" }, - { "prompt": "Détails du ticket JIRA PROJ-12", "expect_action": "ticket-info" }, - { "prompt": "Donne-moi le ticket en cours", "expect_action": "ticket-info" }, - { "prompt": "Que dit le ticket en cours", "expect_action": "ticket-info" }, - { "prompt": "Create a new issue for the auth bug", "expect_action": null }, - { "prompt": "Comment 'looks good' on ticket ABC-123", "expect_action": null }, - { "prompt": "Move ticket ABC-123 to Done", "expect_action": null } -] diff --git a/plugins/aidd-pm/skills/02-user-stories-create/evals/scenarios.json b/plugins/aidd-pm/skills/02-user-stories-create/evals/scenarios.json deleted file mode 100644 index e35a6c33..00000000 --- a/plugins/aidd-pm/skills/02-user-stories-create/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Create user stories for the new search feature", "expect_action": "create-user-stories" }, - { "prompt": "/create-user-stories auth flow", "expect_action": "create-user-stories" }, - { "prompt": "Write INVEST stories for the onboarding redesign", "expect_action": "create-user-stories" }, - { "prompt": "Draft user stories for billing", "expect_action": "create-user-stories" }, - { "prompt": "Generate INVEST user stories for login", "expect_action": "create-user-stories" }, - { "prompt": "Write user stories with acceptance criteria for checkout", "expect_action": "create-user-stories" }, - { "prompt": "User stories en INVEST pour l'auth", "expect_action": "create-user-stories" }, - { "prompt": "Rédige les user stories du panier", "expect_action": "create-user-stories" }, - { "prompt": "Génère des user stories INVEST pour le module search", "expect_action": "create-user-stories" }, - { "prompt": "Écris les histoires utilisateur du checkout", "expect_action": "create-user-stories" }, - { "prompt": "User stories INVEST pour onboarding", "expect_action": "create-user-stories" }, - { "prompt": "Stories utilisateur du module billing", "expect_action": "create-user-stories" }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Draft a PRD for billing", "expect_action": null }, - { "prompt": "Commit my changes", "expect_action": null } -] diff --git a/plugins/aidd-pm/skills/03-prd/evals/scenarios.json b/plugins/aidd-pm/skills/03-prd/evals/scenarios.json deleted file mode 100644 index 36941369..00000000 --- a/plugins/aidd-pm/skills/03-prd/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Draft a PRD for billing", "expect_action": "prd" }, - { "prompt": "/prd auth feature", "expect_action": "prd" }, - { "prompt": "Write product requirements for the search redesign", "expect_action": "prd" }, - { "prompt": "Generate a PRD for the onboarding flow", "expect_action": "prd" }, - { "prompt": "PRD for the new dashboard", "expect_action": "prd" }, - { "prompt": "Make me a product requirements doc for checkout", "expect_action": "prd" }, - { "prompt": "Create a PRD covering scope and acceptance criteria", "expect_action": "prd" }, - { "prompt": "Document product requirements for Stripe integration", "expect_action": "prd" }, - { "prompt": "Rédige un PRD pour le checkout", "expect_action": "prd" }, - { "prompt": "Crée le PRD de la feature login", "expect_action": "prd" }, - { "prompt": "Écris les exigences produit pour le panier", "expect_action": "prd" }, - { "prompt": "Document produit pour le module de recherche", "expect_action": "prd" }, - { "prompt": "Create user stories for billing", "expect_action": null }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Write a technical plan for the auth refactor", "expect_action": null }, - { "prompt": "Draft an architecture diagram for search", "expect_action": null } -] diff --git a/plugins/aidd-pm/skills/04-spec/evals/scenarios.json b/plugins/aidd-pm/skills/04-spec/evals/scenarios.json deleted file mode 100644 index 0830ec5e..00000000 --- a/plugins/aidd-pm/skills/04-spec/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Draft a spec for the billing feature", "expect_action": "build" }, - { "prompt": "/spec for billing", "expect_action": "build" }, - { "prompt": "Generate a spec from the PRD at aidd_docs/tasks/2026_05/billing-prd.md", "expect_action": "build" }, - { "prompt": "Refine the spec to address reviewer findings", "expect_action": "refine" }, - { "prompt": "Generate spec from PRD file", "expect_action": "build" }, - { "prompt": "Build a project spec for checkout", "expect_action": "build" }, - { "prompt": "Spec normalized contract for the auth feature", "expect_action": "build" }, - { "prompt": "Draft spec for login from existing PRD", "expect_action": "build" }, - { "prompt": "Rédige la spec du module billing", "expect_action": "build" }, - { "prompt": "Spec normalisée pour la feature checkout", "expect_action": "build" }, - { "prompt": "Génère la spec depuis le PRD", "expect_action": "build" }, - { "prompt": "Affine la spec selon les retours reviewer", "expect_action": "refine" }, - { "prompt": "Implement the billing feature", "expect_action": null }, - { "prompt": "Draft a PRD for billing", "expect_action": null }, - { "prompt": "Commit my changes", "expect_action": null } -] diff --git a/plugins/aidd-refine/CATALOG.md b/plugins/aidd-refine/CATALOG.md index 393d8a8a..32f3acbe 100644 --- a/plugins/aidd-refine/CATALOG.md +++ b/plugins/aidd-refine/CATALOG.md @@ -42,7 +42,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [04-refine-and-validate.md](skills/01-brainstorm/actions/04-refine-and-validate.md) | - | | `actions` | [05-confirm-approval.md](skills/01-brainstorm/actions/05-confirm-approval.md) | - | | `assets` | [question-templates.md](skills/01-brainstorm/assets/question-templates.md) | - | -| `evals` | [scenarios.json](skills/01-brainstorm/evals/scenarios.json) | - | | `-` | [README.md](skills/01-brainstorm/README.md) | - | | `references` | [ambiguity-detection.md](skills/01-brainstorm/references/ambiguity-detection.md) | - | | `-` | [SKILL.md](skills/01-brainstorm/SKILL.md) | `Interactive brainstorming session to clarify and refine requests through iterative questioning. Use when user mentions unclear requirements, vague ideas, or needs clarification on features. Do NOT use for clear technical specs, implementation details, or when requirements are already well-defined.` | @@ -52,7 +51,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | Group | File | Description | |-------|------|---| | `actions` | [01-challenge.md](skills/02-challenge/actions/01-challenge.md) | - | -| `evals` | [scenarios.json](skills/02-challenge/evals/scenarios.json) | - | | `-` | [README.md](skills/02-challenge/README.md) | - | | `references` | [confidence-rubric.md](skills/02-challenge/references/confidence-rubric.md) | - | | `-` | [SKILL.md](skills/02-challenge/SKILL.md) | `Rethink prior work to verify correctness against an agreed plan, classifying findings as deal-breakers, suggestions, or correct, with a confidence score. Use when the user says "challenge this", "rethink your plan", "is this correct", "review my last decision", "challenge my decision", "challenge what you did", "is my decision right", "criticize this", "find flaws", or asks for a critical review of just-completed work. Do NOT use for line-by-line code review against a style guide, implementing features, writing tests, or generating new code.` | @@ -63,7 +61,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---| | `actions` | [01-condense.md](skills/03-condense/actions/01-condense.md) | - | | `actions` | [02-stats.md](skills/03-condense/actions/02-stats.md) | - | -| `evals` | [scenarios.json](skills/03-condense/evals/scenarios.json) | - | | `-` | [README.md](skills/03-condense/README.md) | - | | `references` | [intensity-levels.md](skills/03-condense/references/intensity-levels.md) | - | | `-` | [SKILL.md](skills/03-condense/SKILL.md) | `Toggle terse output mode with intensity levels (lite, full, ultra) so prose drops articles, filler, and pleasantries while code, quoted errors, and security warnings stay verbatim. Also reports real token usage and estimated savings under condense mode for the current session. Use when the user says "condense", "condense output", "be more concise", "shorter answers", "tighten output", "/condense", "/condense full", "/condense ultra", "stop condense", "normal mode", "/condense-stats", "how much have we saved", or "token savings". Do NOT use for editing existing prose, summarizing a long document, or compressing source code (only output style is affected, not content).` | @@ -76,7 +73,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-render-report.md](skills/04-shadow-areas/actions/02-render-report.md) | - | | `actions` | [03-diff.md](skills/04-shadow-areas/actions/03-diff.md) | - | | `assets` | [report-template.md](skills/04-shadow-areas/assets/report-template.md) | - | -| `evals` | [scenarios.json](skills/04-shadow-areas/evals/scenarios.json) | - | | `-` | [README.md](skills/04-shadow-areas/README.md) | - | | `references` | [categories.md](skills/04-shadow-areas/references/categories.md) | - | | `references` | [locked-sets.json](skills/04-shadow-areas/references/locked-sets.json) | - | @@ -92,7 +88,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [02-verify.md](skills/05-fact-check/actions/02-verify.md) | - | | `actions` | [03-report.md](skills/05-fact-check/actions/03-report.md) | - | | `assets` | [report-template.md](skills/05-fact-check/assets/report-template.md) | - | -| `evals` | [scenarios.json](skills/05-fact-check/evals/scenarios.json) | - | | `-` | [README.md](skills/05-fact-check/README.md) | - | | `references` | [claim-categories.md](skills/05-fact-check/references/claim-categories.md) | - | | `references` | [verification-cascade.md](skills/05-fact-check/references/verification-cascade.md) | - | diff --git a/plugins/aidd-refine/skills/01-brainstorm/evals/scenarios.json b/plugins/aidd-refine/skills/01-brainstorm/evals/scenarios.json deleted file mode 100644 index 11e26e53..00000000 --- a/plugins/aidd-refine/skills/01-brainstorm/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Build a chat app but I'm not sure about the details", "expect_action": "capture-request" }, - { "prompt": "I have an idea for a feature but it's vague", "expect_action": "capture-request" }, - { "prompt": "Need to clarify these requirements", "expect_action": "capture-request" }, - { "prompt": "Brainstorm with me about analytics dashboard", "expect_action": "capture-request" }, - { "prompt": "Help me brainstorm a vague idea", "expect_action": "capture-request" }, - { "prompt": "Brainstorming session to clarify my product idea", "expect_action": "capture-request" }, - { "prompt": "Brainstorm to refine this fuzzy requirement", "expect_action": "capture-request" }, - { "prompt": "Aide-moi à brainstormer cette idée floue", "expect_action": "capture-request" }, - { "prompt": "Brainstorming pour clarifier mon besoin", "expect_action": "capture-request" }, - { "prompt": "Brainstorm une idée d'app", "expect_action": "capture-request" }, - { "prompt": "Affine mon idée par brainstorming", "expect_action": "capture-request" }, - { "prompt": "Brainstorm session pour affiner ces specs floues", "expect_action": "capture-request" }, - { "prompt": "The specs are clear, implement the login feature", "expect_action": null }, - { "prompt": "Write code for user authentication", "expect_action": null }, - { "prompt": "Debug this error in the app", "expect_action": null } -] \ No newline at end of file diff --git a/plugins/aidd-refine/skills/02-challenge/evals/scenarios.json b/plugins/aidd-refine/skills/02-challenge/evals/scenarios.json deleted file mode 100644 index 82deccd3..00000000 --- a/plugins/aidd-refine/skills/02-challenge/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "Challenge what you just did", "expect_action": "challenge" }, - { "prompt": "Is this 100% correct based on what we agreed?", "expect_action": "challenge" }, - { "prompt": "Review my last decision and find deal-breakers", "expect_action": "challenge" }, - { "prompt": "Rethink the plan you proposed", "expect_action": "challenge" }, - { "prompt": "Find flaws in this approach", "expect_action": "challenge" }, - { "prompt": "Criticize the design and propose deal-breakers", "expect_action": "challenge" }, - { "prompt": "Challenge this decision", "expect_action": "challenge" }, - { "prompt": "/challenge this work", "expect_action": "challenge" }, - { "prompt": "Remets en cause cette décision", "expect_action": "challenge" }, - { "prompt": "Critique ce plan et trouve les failles", "expect_action": "challenge" }, - { "prompt": "Challenge ma décision sur ce design", "expect_action": "challenge" }, - { "prompt": "Trouve les deal-breakers dans cette approche", "expect_action": "challenge" }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Write a unit test for parseUser", "expect_action": null }, - { "prompt": "Generate a Mermaid diagram of the data flow", "expect_action": null } -] diff --git a/plugins/aidd-refine/skills/03-condense/evals/scenarios.json b/plugins/aidd-refine/skills/03-condense/evals/scenarios.json deleted file mode 100644 index b3365c2a..00000000 --- a/plugins/aidd-refine/skills/03-condense/evals/scenarios.json +++ /dev/null @@ -1,20 +0,0 @@ -[ - { "prompt": "Condense your output", "expect_action": "condense" }, - { "prompt": "Be more concise from now on", "expect_action": "condense" }, - { "prompt": "/condense full", "expect_action": "condense" }, - { "prompt": "/condense ultra", "expect_action": "condense" }, - { "prompt": "/condense lite", "expect_action": "condense" }, - { "prompt": "Stop condense", "expect_action": "condense" }, - { "prompt": "Normal mode", "expect_action": "condense" }, - { "prompt": "Tighten output", "expect_action": "condense" }, - { "prompt": "Shorter answers please", "expect_action": "condense" }, - { "prompt": "/condense-stats", "expect_action": "stats" }, - { "prompt": "How much have we saved this session?", "expect_action": "stats" }, - { "prompt": "Show me the token savings", "expect_action": "stats" }, - { "prompt": "Condense stats", "expect_action": "stats" }, - { "prompt": "Explain in detail how React reconciliation works", "expect_action": null }, - { "prompt": "Write a long README for this repo", "expect_action": null }, - { "prompt": "Implement the auth feature", "expect_action": null }, - { "prompt": "Summarize this long document into one paragraph", "expect_action": null }, - { "prompt": "Compress this Python source file", "expect_action": null } -] diff --git a/plugins/aidd-refine/skills/04-shadow-areas/evals/scenarios.json b/plugins/aidd-refine/skills/04-shadow-areas/evals/scenarios.json deleted file mode 100644 index 61f7bcdf..00000000 --- a/plugins/aidd-refine/skills/04-shadow-areas/evals/scenarios.json +++ /dev/null @@ -1,25 +0,0 @@ -[ - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/foo-prd.md", "expect_action": "shadow-areas" }, - { "prompt": "find blind spots in this spec", "expect_action": "shadow-areas" }, - { "prompt": "what's missing in this PRD", "expect_action": "shadow-areas" }, - { "prompt": "shadow report on the spec", "expect_action": "shadow-areas" }, - { "prompt": "shadow analysis of the user stories", "expect_action": "shadow-areas" }, - { "prompt": "scan for gaps in the requirements", "expect_action": "shadow-areas" }, - { "prompt": "find what's missing in the auth PRD", "expect_action": "shadow-areas" }, - { "prompt": "spot blind spots in the design", "expect_action": "shadow-areas" }, - { "prompt": "review for gaps the checkout spec", "expect_action": "shadow-areas" }, - { "prompt": "analytical gap scan of this artifact", "expect_action": "shadow-areas" }, - { "prompt": "trouve les blind spots dans ce PRD", "expect_action": "shadow-areas" }, - { "prompt": "scanne les gaps de la spec", "expect_action": "shadow-areas" }, - { "prompt": "qu'est-ce qui manque dans ce PRD", "expect_action": "shadow-areas" }, - { "prompt": "rapport shadow sur la spec", "expect_action": "shadow-areas" }, - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/empty-file.md", "expect_action": "shadow-areas" }, - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/diagram.png", "expect_action": "shadow-areas" }, - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/giant-spec.md", "expect_action": "shadow-areas" }, - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/new-prd.md with prior report aidd_docs/tasks/2026_05/new-prd-shadow-report.md", "expect_action": "shadow-areas" }, - { "prompt": "run shadow-areas on aidd_docs/tasks/nonexistent/missing-file.md", "expect_action": null }, - { "prompt": "run shadow-areas on aidd_docs/tasks/2026_05/new-prd-shadow-report.md", "expect_action": null }, - { "prompt": "brainstorm a new feature idea", "expect_action": null }, - { "prompt": "implement the auth feature", "expect_action": null }, - { "prompt": "commit my changes", "expect_action": null } -] diff --git a/plugins/aidd-refine/skills/05-fact-check/evals/scenarios.json b/plugins/aidd-refine/skills/05-fact-check/evals/scenarios.json deleted file mode 100644 index 6de7a67a..00000000 --- a/plugins/aidd-refine/skills/05-fact-check/evals/scenarios.json +++ /dev/null @@ -1,17 +0,0 @@ -[ - { "prompt": "fact-check this", "expect_action": "fact-check" }, - { "prompt": "verify that claim", "expect_action": "fact-check" }, - { "prompt": "are you sure about that?", "expect_action": "fact-check" }, - { "prompt": "is that actually true?", "expect_action": "fact-check" }, - { "prompt": "cite your sources for this", "expect_action": "fact-check" }, - { "prompt": "where did you get that fact?", "expect_action": "fact-check" }, - { "prompt": "did you make that up?", "expect_action": "fact-check" }, - { "prompt": "double-check the version number you gave me", "expect_action": "fact-check" }, - { "prompt": "vérifie cette information", "expect_action": "fact-check" }, - { "prompt": "es-tu sûr de ça ?", "expect_action": "fact-check" }, - { "prompt": "implement the login form", "expect_action": null }, - { "prompt": "commit my changes", "expect_action": null }, - { "prompt": "what do you think of this design?", "expect_action": null }, - { "prompt": "rename this variable to camelCase", "expect_action": null }, - { "prompt": "refactor this function", "expect_action": null } -] diff --git a/plugins/aidd-vcs/CATALOG.md b/plugins/aidd-vcs/CATALOG.md index 1cc179a4..4cc8de8d 100644 --- a/plugins/aidd-vcs/CATALOG.md +++ b/plugins/aidd-vcs/CATALOG.md @@ -29,7 +29,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---|---| | `actions` | [01-commit.md](skills/01-commit/actions/01-commit.md) | - | - | | `assets` | [commit-template.md](skills/01-commit/assets/commit-template.md) | `VCS commit message template` | - | -| `evals` | [scenarios.json](skills/01-commit/evals/scenarios.json) | - | - | | `-` | [README.md](skills/01-commit/README.md) | - | - | | `-` | [SKILL.md](skills/01-commit/SKILL.md) | `Create an atomic git commit with conventional message format. Use when the user says "commit", "git commit", "create a commit", "commit my changes", "commit and push", or invokes `/commit`. Do NOT use for amending existing commits, force-pushing, rebasing, opening pull requests, or release tagging.` | - | @@ -42,7 +41,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `assets` | [CONTRIBUTING.md](skills/02-pull-request/assets/CONTRIBUTING.md) | `Project contribution guidelines template` | - | | `assets` | [pull_request.md](skills/02-pull-request/assets/pull_request.md) | `VCS pull/merge request template` | - | | `assets` | [README.md](skills/02-pull-request/assets/README.md) | `Project README template` | - | -| `evals` | [scenarios.json](skills/02-pull-request/evals/scenarios.json) | - | - | | `-` | [README.md](skills/02-pull-request/README.md) | - | - | | `-` | [SKILL.md](skills/02-pull-request/SKILL.md) | `Create a draft pull or merge request from the current branch. Use when the user says "open a pr", "open a pull request", "create a pr", "create a merge request", "open mr", "draft a pr for this branch", or invokes `/pull-request`. Do NOT use for committing changes, pushing a branch directly, tagging releases, merging an existing request, or amending commits.` | - | @@ -52,7 +50,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---|---| | `actions` | [01-release-tag.md](skills/03-release-tag/actions/01-release-tag.md) | - | - | | `assets` | [release-template.md](skills/03-release-tag/assets/release-template.md) | `VCS release notes template` | - | -| `evals` | [scenarios.json](skills/03-release-tag/evals/scenarios.json) | - | - | | `-` | [README.md](skills/03-release-tag/README.md) | - | - | | `-` | [SKILL.md](skills/03-release-tag/SKILL.md) | `Cut a semver release with annotated tag and release notes. Use when the user says "release", "tag", "tag this release", "bump version", "release v1.2.0", "cut a release", or invokes `/release-tag`. Do NOT use for plain commits without a tag, opening pull requests, pushing a branch only, or amending existing tags.` | - | @@ -63,7 +60,6 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [01-issue-create.md](skills/04-issue-create/actions/01-issue-create.md) | - | - | | `assets` | [CONTRIBUTING.md](skills/04-issue-create/assets/CONTRIBUTING.md) | `Project contribution guidelines template` | - | | `assets` | [issue-template.md](skills/04-issue-create/assets/issue-template.md) | `VCS issue/ticket template` | - | -| `evals` | [scenarios.json](skills/04-issue-create/evals/scenarios.json) | - | - | | `-` | [README.md](skills/04-issue-create/README.md) | - | - | | `-` | [SKILL.md](skills/04-issue-create/SKILL.md) | `Create an issue in the configured ticketing tool. Use when the user says "new issue", "create an issue", "file a bug", "file an issue", "report bug", "open an issue", or invokes `/issue-create`. Do NOT use for committing changes, opening pull requests, tagging releases, or commenting on existing issues.` | - | diff --git a/plugins/aidd-vcs/skills/01-commit/evals/scenarios.json b/plugins/aidd-vcs/skills/01-commit/evals/scenarios.json deleted file mode 100644 index 6ddb9793..00000000 --- a/plugins/aidd-vcs/skills/01-commit/evals/scenarios.json +++ /dev/null @@ -1,24 +0,0 @@ -[ - { "prompt": "Commit my changes", "expect_action": "commit" }, - { "prompt": "/commit", "expect_action": "commit" }, - { "prompt": "Create a commit with message 'fix typo'", "expect_action": "commit" }, - { "prompt": "Commit and push", "expect_action": "commit" }, - { "prompt": "Save my work", "expect_action": "commit" }, - { "prompt": "Snapshot this state", "expect_action": "commit" }, - { "prompt": "Record this change", "expect_action": "commit" }, - { "prompt": "git commit -m 'wip'", "expect_action": "commit" }, - { "prompt": "Stage and commit", "expect_action": "commit" }, - { "prompt": "Valider mes changements", "expect_action": "commit" }, - { "prompt": "Enregistre les modifications", "expect_action": "commit" }, - { "prompt": "Commit ce que j'ai fait", "expect_action": "commit" }, - { "prompt": "Ship this commit", "expect_action": "commit" }, - { "prompt": "Publish my commit", "expect_action": "commit" }, - { "prompt": "Fige cette version", "expect_action": "commit" }, - { "prompt": "Push the current branch to origin", "expect_action": null }, - { "prompt": "Open a pull request", "expect_action": null }, - { "prompt": "Tag this release v1.2.0", "expect_action": null }, - { "prompt": "Cherry-pick that commit onto main", "expect_action": null }, - { "prompt": "Revert the last commit", "expect_action": null }, - { "prompt": "Explain this code", "expect_action": null }, - { "prompt": "Explique-moi le code", "expect_action": null } -] diff --git a/plugins/aidd-vcs/skills/02-pull-request/evals/scenarios.json b/plugins/aidd-vcs/skills/02-pull-request/evals/scenarios.json deleted file mode 100644 index 45b4c03c..00000000 --- a/plugins/aidd-vcs/skills/02-pull-request/evals/scenarios.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { "prompt": "Open a pull request", "expect_action": "pull-request" }, - { "prompt": "/pull-request", "expect_action": "pull-request" }, - { "prompt": "Create a merge request from this branch", "expect_action": "pull-request" }, - { "prompt": "Draft a PR for the current branch", "expect_action": "pull-request" }, - { "prompt": "Open PR", "expect_action": "pull-request" }, - { "prompt": "Submit a merge request", "expect_action": "pull-request" }, - { "prompt": "Make a draft pull request", "expect_action": "pull-request" }, - { "prompt": "Open MR", "expect_action": "pull-request" }, - { "prompt": "Ouvre une PR", "expect_action": "pull-request" }, - { "prompt": "Crée une merge request", "expect_action": "pull-request" }, - { "prompt": "Demande de merge sur main", "expect_action": "pull-request" }, - { "prompt": "Soumets une pull request", "expect_action": "pull-request" }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Push the current branch", "expect_action": null }, - { "prompt": "Tag this release v1.2.0", "expect_action": null }, - { "prompt": "Review this pull request", "expect_action": null }, - { "prompt": "Merge that pull request now", "expect_action": null } -] diff --git a/plugins/aidd-vcs/skills/03-release-tag/evals/scenarios.json b/plugins/aidd-vcs/skills/03-release-tag/evals/scenarios.json deleted file mode 100644 index 2c1ccb3b..00000000 --- a/plugins/aidd-vcs/skills/03-release-tag/evals/scenarios.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - { "prompt": "Tag this release", "expect_action": "release-tag" }, - { "prompt": "/release-tag", "expect_action": "release-tag" }, - { "prompt": "Bump to v2.0.0", "expect_action": "release-tag" }, - { "prompt": "Cut a release v1.2.0", "expect_action": "release-tag" }, - { "prompt": "Tag v1.0.0", "expect_action": "release-tag" }, - { "prompt": "Bump version major", "expect_action": "release-tag" }, - { "prompt": "Release version 3", "expect_action": "release-tag" }, - { "prompt": "Publish v2.0", "expect_action": "release-tag" }, - { "prompt": "Crée le tag v2.1", "expect_action": "release-tag" }, - { "prompt": "Marque la release v1.5", "expect_action": "release-tag" }, - { "prompt": "Sort la version 4.0", "expect_action": "release-tag" }, - { "prompt": "Bump le numéro de version", "expect_action": "release-tag" }, - { "prompt": "Fais une release", "expect_action": "release-tag" }, - { "prompt": "Do a release", "expect_action": "release-tag" }, - { "prompt": "Make release", "expect_action": "release-tag" }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Open a pull request", "expect_action": null }, - { "prompt": "Push the current branch", "expect_action": null }, - { "prompt": "Show the latest release notes", "expect_action": null }, - { "prompt": "Delete the v1.0 tag", "expect_action": null } -] diff --git a/plugins/aidd-vcs/skills/04-issue-create/evals/scenarios.json b/plugins/aidd-vcs/skills/04-issue-create/evals/scenarios.json deleted file mode 100644 index 5000ed7f..00000000 --- a/plugins/aidd-vcs/skills/04-issue-create/evals/scenarios.json +++ /dev/null @@ -1,20 +0,0 @@ -[ - { "prompt": "File a bug for the auth flow", "expect_action": "issue-create" }, - { "prompt": "/issue-create", "expect_action": "issue-create" }, - { "prompt": "Create an issue: login fails on Safari", "expect_action": "issue-create" }, - { "prompt": "Open an issue in the tracker for this regression", "expect_action": "issue-create" }, - { "prompt": "Report a bug: checkout returns 500", "expect_action": "issue-create" }, - { "prompt": "Log a defect for the payment crash", "expect_action": "issue-create" }, - { "prompt": "File issue in tracker for the auth bug", "expect_action": "issue-create" }, - { "prompt": "Open a GitHub issue for this regression", "expect_action": "issue-create" }, - { "prompt": "Déclare un bug dans le tracker", "expect_action": "issue-create" }, - { "prompt": "Crée une issue GitHub pour ce crash", "expect_action": "issue-create" }, - { "prompt": "Ouvre un ticket bug pour le checkout", "expect_action": "issue-create" }, - { "prompt": "Rapporte un défaut dans le tracker", "expect_action": "issue-create" }, - { "prompt": "Commit my changes", "expect_action": null }, - { "prompt": "Open a pull request", "expect_action": null }, - { "prompt": "Tag this release v1.2.0", "expect_action": null }, - { "prompt": "Show ticket info for ABC-123", "expect_action": null }, - { "prompt": "Comment on this issue", "expect_action": null } -] -