From 030a6cb22887e7a8082cebb9dda5edaa45338d86 Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Thu, 11 Jun 2026 10:47:04 +1000 Subject: [PATCH 1/2] chore: add activity-coverage dimension to gap-analysis skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a sixth audit dimension (and an `activities` invocation scope) that diffs which of the 7 core activity kinds each provider ships an adapter for against what upstream offers, with a provider×activity grid in the report template. Co-Authored-By: Claude Fable 5 --- .claude/skills/gap-analysis/SKILL.md | 54 ++++++++++--- .../references/audit-checklist.md | 76 +++++++++++++++++-- .../references/report-template.md | 23 ++++++ 3 files changed, 137 insertions(+), 16 deletions(-) diff --git a/.claude/skills/gap-analysis/SKILL.md b/.claude/skills/gap-analysis/SKILL.md index 0178a00c3..eb144b195 100644 --- a/.claude/skills/gap-analysis/SKILL.md +++ b/.claude/skills/gap-analysis/SKILL.md @@ -15,13 +15,16 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** ## Invocation -| Args | Scope | -| -------------------------------- | --------------------------------------------------- | -| `` (e.g. `openai`) | One provider — all four audit dimensions. | -| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | -| `models` | New-model diff for every provider. | -| `--all` | Full sweep (fan out subagents, one per provider). | -| _(none)_ | Ask the user which scope via AskUserQuestion. | +| Args | Scope | +| -------------------------------- | ------------------------------------------------------ | +| `` (e.g. `openai`) | One provider — all audit dimensions. | +| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | +| `models` | New-model diff for every provider. | +| `activities` | Activity-coverage diff: which of the 7 core activity | +| | kinds each provider ships an adapter for, vs. what | +| | upstream supports. (Dimension 6 only, all providers.) | +| `--all` | Full sweep (fan out subagents, one per provider). | +| _(none)_ | Ask the user which scope via AskUserQuestion. | ## Workflow @@ -44,12 +47,18 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** 4. Capability-flag drift 5. Telemetry / observability parity (usage tokens, cache/reasoning counts, request ids, logging asymmetry) + 6. Activity coverage (which of the 7 core activity kinds each provider + ships an adapter for vs. what upstream supports) — this is the **only** + dimension for the `activities` scope; it's also rolled into `--all`. 5. **Fan out** for `--all`: launch one `Explore` subagent per provider, max 3 - in parallel. Each subagent returns the five-dimension findings for its - provider; you synthesise into the combined report. + in parallel. Each subagent returns the multi-dimension findings for its + provider; you synthesise into the combined report. The `activities` scope + does **not** fan out — derive the provider×activity matrix centrally from + the adapter files (see dimension 6), since it's a fast mechanical diff. 6. **Write the report** to `.agent/gap-analysis/YYYY-MM-DD-.md` using [references/report-template.md](references/report-template.md). Date is - today's ISO date. `` is `openai` / `feature-tts` / `models` / `all`. + today's ISO date. `` is `openai` / `feature-tts` / `models` / + `activities` / `all`. 7. **Print the report path and a 5-line summary** to the user. ## Critical rules @@ -88,6 +97,31 @@ re-read it; this list is a snapshot: `multimodal-structured`, `summarize`, `summarize-stream`, `image-gen`, `tts`, `transcription`, `video-gen`. +## Known activities (7) + +**Features** (above) are matrix rows about behaviours within an activity. +**Activities** are the coarser-grained core capability kinds in `@tanstack/ai` +— each has a `BaseAdapter` and a provider "supports" one only if its +package ships an adapter of that kind. Canonical list is the `AdapterKind` +union in `packages/ai/src/activities/index.ts` — always re-read it: + +`text`, `summarize`, `image`, `audio`, `video`, `tts`, `transcription`. + +A provider's activity surface is derived mechanically from its adapter files: +`packages/ai-/src/adapters/`. Filename → activity-kind map: + +| Adapter file | Activity kind | +| ------------------------------------- | --------------- | +| `text.ts` / `text-chat-completions.ts` / `responses-text.ts` | `text` | +| `summarize.ts` | `summarize` | +| `image.ts` | `image` | +| `audio.ts` | `audio` | +| `video.ts` | `video` | +| `speech.ts` / `tts.ts` | `tts` | +| `transcription.ts` | `transcription` | + +(`cost.ts` is a helper, not an activity adapter.) + ## Verification before finishing Before printing the summary: diff --git a/.claude/skills/gap-analysis/references/audit-checklist.md b/.claude/skills/gap-analysis/references/audit-checklist.md index 36a4cf134..6ad6fccff 100644 --- a/.claude/skills/gap-analysis/references/audit-checklist.md +++ b/.claude/skills/gap-analysis/references/audit-checklist.md @@ -217,6 +217,65 @@ Steps: --- +## 6. Activity coverage + +**Input:** each provider's `packages/ai-/src/adapters/` directory. +**Reference:** the `AdapterKind` union in `packages/ai/src/activities/index.ts` +(7 kinds: `text`, `summarize`, `image`, `audio`, `video`, `tts`, +`transcription`). +**Upstream:** each provider's API reference / models page. + +This dimension answers a coarser question than dimension 2: not "which +behaviour within chat is exercised" but **"which whole activity kinds does the +provider's API offer that we ship no adapter for at all."** + +Steps: + +1. For every provider, list its adapter files and map each to an activity kind + using the filename→kind table in `SKILL.md` (§ Known activities). That + yields the local provider×activity matrix. `text.ts`, + `text-chat-completions.ts`, and `responses-text.ts` all count as `text`; + `speech.ts` and `tts.ts` both count as `tts`; ignore `cost.ts` and other + helpers. +2. Build the full grid (rows = 9 providers, cols = 7 activity kinds). A cell is + ✅ if an adapter file of that kind exists, ❌ otherwise. +3. For each ❌ cell, WebFetch the provider's API reference and check whether + upstream offers that activity: + - **Real gap** — upstream offers it, we ship no adapter, and there is no + documented reason. High if it's a flagship activity for that provider + (e.g. a TTS provider missing `transcription`), medium otherwise. + - **Not a gap** — upstream genuinely doesn't offer that activity (e.g. + Anthropic has no image/audio/video/tts/transcription endpoints; Groq has + no image/video generation). Mark the cell ❌-by-design and omit from the + gap list, but keep it in the grid so the report is self-contained. +4. Note the inverse too: any **local adapter for an activity the provider no + longer offers upstream** (rare; surface as a stale-adapter finding). + +Map activity kind → upstream capability to look for: + +| Activity kind | Upstream capability | +| --------------- | ---------------------------------------------------- | +| `text` | Chat / messages / completions endpoint | +| `summarize` | Any text completion (summarize is built on chat) | +| `image` | Image generation endpoint | +| `audio` | Music / sound / general audio generation endpoint | +| `video` | Video generation endpoint | +| `tts` | Text-to-speech endpoint | +| `transcription` | Speech-to-text endpoint | + +**Priority rubric:** + +- Missing a core activity the provider's API clearly offers (e.g. OpenAI-class + provider with no `image`/`tts`/`transcription`) → **high**. +- Missing a secondary/media activity → **medium**. +- ❌-by-design (upstream doesn't offer it) → **out-of-scope**, keep in grid only. +- Local adapter for a now-removed upstream activity → **medium** (deprecate). + +Render the grid in the report under its own "Activity coverage" section, with +one bullet per real gap citing the upstream URL. + +--- + ## Subagent dispatch (for `--all` scope) When fan-out is needed, launch one `Explore` subagent per provider with a @@ -225,12 +284,17 @@ prompt of this shape: > Audit the `` adapter at `packages/ai-/` > against upstream docs at the URLs in > `.claude/skills/gap-analysis/references/provider-doc-urls.md`. Walk -> dimensions 1, 3, 4, and 5 from `audit-checklist.md`. Skip dimension 2 -> (the orchestrator handles cross-provider parity centrally) — but do -> emit dimension-5 telemetry rows in the per-provider format; the -> orchestrator stitches them into the cross-adapter table. Return -> findings as markdown sections matching the report template — High / -> Medium / Low / Out-of-scope — with upstream URLs cited for every claim. +> dimensions 1, 3, 4, and 5 from `audit-checklist.md`. Skip dimensions 2 +> and 6 (the orchestrator handles cross-provider parity and the +> activity-coverage grid centrally) — but do emit dimension-5 telemetry +> rows in the per-provider format; the orchestrator stitches them into the +> cross-adapter table. Return findings as markdown sections matching the +> report template — High / Medium / Low / Out-of-scope — with upstream +> URLs cited for every claim. + +The orchestrator builds the dimension-6 activity-coverage grid itself from +the adapter-file listing (a one-shot `ls packages/ai-*/src/adapters/`), then +WebFetches each provider's API reference only for the ❌ cells. Run at most 3 in parallel. Aggregate their returned markdown into the combined report. diff --git a/.claude/skills/gap-analysis/references/report-template.md b/.claude/skills/gap-analysis/references/report-template.md index 3c815635a..96c708d69 100644 --- a/.claude/skills/gap-analysis/references/report-template.md +++ b/.claude/skills/gap-analysis/references/report-template.md @@ -95,6 +95,29 @@ Replace every `{{placeholder}}`. Drop sections that have zero entries --- +## Activity coverage + +> Per dimension 6 in audit-checklist.md. Which of the 7 core activity kinds +> (`text`, `summarize`, `image`, `audio`, `video`, `tts`, `transcription`) +> each provider ships an adapter for, vs. what upstream offers. Derived from +> `packages/ai-/src/adapters/`. + +| Provider | text | summarize | image | audio | video | tts | transcription | +| -------- | :--: | :-------: | :---: | :---: | :---: | :-: | :-----------: | +| {{provider}} | {{✅/❌/—}} | … | … | … | … | … | … | + +Legend: ✅ adapter shipped · ❌ upstream offers it, no adapter (gap) · — upstream doesn't offer it (by design). + +- **[{{provider}}] missing `{{activity}}` activity** + - Upstream: [{{doc-title}}]({{doc-url}}) + - Current state: no `{{file}}.ts` under `packages/ai-{{provider}}/src/adapters/` + - Suggested change: add a `{{provider}}{{Activity}}` adapter (mirror {{sibling provider that has it}}). + - Effort: {{S / M / L}} + +{{repeat per real activity gap}} + +--- + ## Out-of-scope — documented exclusions > Listed for completeness; no action required. Each links to the comment From 10c8ebb60974b8ce6a77f2501fafdcfbe5caa271 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Thu, 11 Jun 2026 00:48:36 +0000 Subject: [PATCH 2/2] ci: apply automated fixes --- .claude/skills/gap-analysis/SKILL.md | 38 +++++++++---------- .../references/audit-checklist.md | 18 ++++----- .../references/report-template.md | 6 +-- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/.claude/skills/gap-analysis/SKILL.md b/.claude/skills/gap-analysis/SKILL.md index eb144b195..7efcc2005 100644 --- a/.claude/skills/gap-analysis/SKILL.md +++ b/.claude/skills/gap-analysis/SKILL.md @@ -15,16 +15,16 @@ markdown report under `.agent/gap-analysis/`. **Do not edit source files.** ## Invocation -| Args | Scope | -| -------------------------------- | ------------------------------------------------------ | -| `` (e.g. `openai`) | One provider — all audit dimensions. | -| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | -| `models` | New-model diff for every provider. | -| `activities` | Activity-coverage diff: which of the 7 core activity | -| | kinds each provider ships an adapter for, vs. what | -| | upstream supports. (Dimension 6 only, all providers.) | -| `--all` | Full sweep (fan out subagents, one per provider). | -| _(none)_ | Ask the user which scope via AskUserQuestion. | +| Args | Scope | +| -------------------------------- | ----------------------------------------------------- | +| `` (e.g. `openai`) | One provider — all audit dimensions. | +| `feature ` (e.g. `tts`) | One feature row of the matrix across all providers. | +| `models` | New-model diff for every provider. | +| `activities` | Activity-coverage diff: which of the 7 core activity | +| | kinds each provider ships an adapter for, vs. what | +| | upstream supports. (Dimension 6 only, all providers.) | +| `--all` | Full sweep (fan out subagents, one per provider). | +| _(none)_ | Ask the user which scope via AskUserQuestion. | ## Workflow @@ -110,15 +110,15 @@ union in `packages/ai/src/activities/index.ts` — always re-read it: A provider's activity surface is derived mechanically from its adapter files: `packages/ai-/src/adapters/`. Filename → activity-kind map: -| Adapter file | Activity kind | -| ------------------------------------- | --------------- | -| `text.ts` / `text-chat-completions.ts` / `responses-text.ts` | `text` | -| `summarize.ts` | `summarize` | -| `image.ts` | `image` | -| `audio.ts` | `audio` | -| `video.ts` | `video` | -| `speech.ts` / `tts.ts` | `tts` | -| `transcription.ts` | `transcription` | +| Adapter file | Activity kind | +| ------------------------------------------------------------ | --------------- | +| `text.ts` / `text-chat-completions.ts` / `responses-text.ts` | `text` | +| `summarize.ts` | `summarize` | +| `image.ts` | `image` | +| `audio.ts` | `audio` | +| `video.ts` | `video` | +| `speech.ts` / `tts.ts` | `tts` | +| `transcription.ts` | `transcription` | (`cost.ts` is a helper, not an activity adapter.) diff --git a/.claude/skills/gap-analysis/references/audit-checklist.md b/.claude/skills/gap-analysis/references/audit-checklist.md index 6ad6fccff..4508c69e3 100644 --- a/.claude/skills/gap-analysis/references/audit-checklist.md +++ b/.claude/skills/gap-analysis/references/audit-checklist.md @@ -253,15 +253,15 @@ Steps: Map activity kind → upstream capability to look for: -| Activity kind | Upstream capability | -| --------------- | ---------------------------------------------------- | -| `text` | Chat / messages / completions endpoint | -| `summarize` | Any text completion (summarize is built on chat) | -| `image` | Image generation endpoint | -| `audio` | Music / sound / general audio generation endpoint | -| `video` | Video generation endpoint | -| `tts` | Text-to-speech endpoint | -| `transcription` | Speech-to-text endpoint | +| Activity kind | Upstream capability | +| --------------- | ------------------------------------------------- | +| `text` | Chat / messages / completions endpoint | +| `summarize` | Any text completion (summarize is built on chat) | +| `image` | Image generation endpoint | +| `audio` | Music / sound / general audio generation endpoint | +| `video` | Video generation endpoint | +| `tts` | Text-to-speech endpoint | +| `transcription` | Speech-to-text endpoint | **Priority rubric:** diff --git a/.claude/skills/gap-analysis/references/report-template.md b/.claude/skills/gap-analysis/references/report-template.md index 96c708d69..05546d245 100644 --- a/.claude/skills/gap-analysis/references/report-template.md +++ b/.claude/skills/gap-analysis/references/report-template.md @@ -102,9 +102,9 @@ Replace every `{{placeholder}}`. Drop sections that have zero entries > each provider ships an adapter for, vs. what upstream offers. Derived from > `packages/ai-/src/adapters/`. -| Provider | text | summarize | image | audio | video | tts | transcription | -| -------- | :--: | :-------: | :---: | :---: | :---: | :-: | :-----------: | -| {{provider}} | {{✅/❌/—}} | … | … | … | … | … | … | +| Provider | text | summarize | image | audio | video | tts | transcription | +| ------------ | :---------: | :-------: | :---: | :---: | :---: | :-: | :-----------: | +| {{provider}} | {{✅/❌/—}} | … | … | … | … | … | … | Legend: ✅ adapter shipped · ❌ upstream offers it, no adapter (gap) · — upstream doesn't offer it (by design).