Skip to content

feat(ai-openrouter): video generation adapter (/api/v1/videos) + image activity follow-ups #707

Description

@tombeckenham

Summary

Follow-up to #618 / #624. OpenRouter has a dedicated async video generation API (POST /api/v1/videos) with Seedance 2.0, Veo 3.1, Wan and others — separate from the chat-completions pathway our current @tanstack/ai-openrouter image adapter uses. We should add a video adapter, and close a few image-activity gaps surfaced while reviewing #624.

1. New: openrouterVideo adapter (generateVideo activity)

OpenRouter's video API (docs, announcement):

  • POST https://openrouter.ai/api/v1/videos202 with polling_url; poll GET /api/v1/videos/{jobId} until completed/failed; download from unsigned_urls[0]. Optional webhooks (video.generation.completed/failed/..., HMAC-SHA256).
  • Request fields: model, prompt, duration, resolution, aspect_ratio, size, seed, generate_audio, callback_url, provider (passthrough).

The media-input fields map 1:1 onto the MediaInputRole taxonomy from #624:

TanStack imageInputs role OpenRouter field
start_frame frame_images[] with frame_type: "first_frame"
end_frame frame_images[] with frame_type: "last_frame"
reference / character input_references[]

(If both frame_images and input_references are present, OpenRouter treats the request as image-to-video; references take lower priority — i2v cookbook, reference-to-video cookbook.)

Implementation notes:

  • Async job lifecycle is closest to the OpenAI Sora adapter (packages/ai-openai/src/adapters/video.ts) — submit + poll, streaming progress via the existing video activity job path.
  • Video models do not appear in plain GET /api/v1/models; use GET /api/v1/videos/models (or ?output_modalities=video). The model-meta sync script needs to learn this second endpoint.
  • Image inputs should pass URLs through verbatim and convert data sources to data URIs (same policy as the image adapter — no server-side fetching). Note OpenRouter warns providers can't fetch URLs behind redirects/bot checks; surface that in docs.
  • videoInputs / audioInputs: not supported by the API today → keep the throw-on-nonempty convention from feat: multimodal prompt for generateImage/generateVideo (image-to-image, image-to-video) #624.

2. Image activity gaps (chat-completions pathway)

From the #624 review:

  • Unmapped size is silently droppedSIZE_TO_ASPECT_RATIO is a fixed 10-entry table; any other size resolves to undefined and no aspect ratio is sent (packages/ai-openrouter/src/adapters/image.ts). Warn or throw instead.
  • Verify imageConfig key casing against the SDK/API contract — the request mixes numberOfImages (camelCase) with aspect_ratio / image_size (snake_case), and relies on OpenRouter "filtering invalid config", which would mask a miskeyed field as a permanent silent no-op. The HTTP API documents image_config with snake_case fields.
  • image_config.strength (0.0–1.0 image-to-image influence) is documented for i2i models — consider exposing it via provider options now that imageInputs lands in feat: multimodal prompt for generateImage/generateVideo (image-to-image, image-to-video) #624.

Definition of done

Per repo couplings (CLAUDE.md + .agent/self-learning/coupling.json), the implementing PR must include:

  • openrouterVideo adapter + model metadata (incl. sync-script support for GET /api/v1/videos/models)
  • imageInputs role mapping per the table above; throws for unsupported roles/cardinalities
  • Image-adapter fixes (size handling, imageConfig casing verification, optional strength)
  • Unit tests for adapter request shapes + polling lifecycle
  • E2E: feature-support matrix entries (and spec/fixture if aimock can mock the async job endpoints — same constraint noted in feat: multimodal prompt for generateImage/generateVideo (image-to-image, image-to-video) #624)
  • Docs: docs/media/video-generation.md provider table + docs/adapters/openrouter.md
  • Skill: packages/ai/skills/ai-core/media-generation/SKILL.md provider table

Refs: #618, #624

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions