Skip to content

feat: add vision model preprocessor with persistent media cache - #2056

Open
StepanchukYI wants to merge 2 commits into
HKUDS:mainfrom
StepanchukYI:pr/vision-media-cache
Open

feat: add vision model preprocessor with persistent media cache#2056
StepanchukYI wants to merge 2 commits into
HKUDS:mainfrom
StepanchukYI:pr/vision-media-cache

Conversation

@StepanchukYI

Copy link
Copy Markdown

Summary

  • Adds vision model support as a preprocessing step: images in messages are transcribed by a vision-capable model before being sent to the main agent LLM
  • Introduces MediaCache — a persistent SHA-256 keyed JSON cache that avoids re-transcribing the same images
  • Configurable via vision_model in agent defaults (e.g. vision_model: "openai/gpt-4o")
  • Auto-detects vision-capable models by pattern matching (gpt-4o, claude-3, gemini, etc.)

How it works

  1. When a message contains images and the main model doesn't support vision, the vision model transcribes them
  2. Transcriptions are cached by SHA-256 hash of the image content
  3. The text description replaces the image in the message sent to the main model

Test plan

  • Existing tests pass (339/339)
  • Verify vision preprocessing works with image messages
  • Verify cache persists and avoids redundant API calls
  • Verify pass-through when main model supports vision natively

🤖 Generated with Claude Code

When a text-only model is configured, use a vision-capable model as
a fallback preprocessor to transcribe images. Transcriptions are
cached persistently in MediaCache to avoid redundant API calls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@StepanchukYI
StepanchukYI force-pushed the pr/vision-media-cache branch from 2927389 to 64be7e7 Compare March 21, 2026 13:54
Sync with upstream v0.1.5, resolve conflicts keeping both
upstream and vision-media-cache features.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants