Skip to content

Commit 7f88cc1

Browse files
CSCSoftwareclaude
andcommitted
feat: Add screenshot optimization — scale + color reduction (up to 95% smaller)
New parameters `scale` (0.1-1.0) and `colors` (2/4/16/256) for aidex_screenshot. Reduces file size dramatically for LLM context (e.g., 108 KB → 5 KB with scale=0.5, colors=2). Cross-platform post-processing: Windows (System.Drawing), macOS (sips), Linux (ImageMagick). Tool description includes iterative strategy for AI assistants to auto-optimize. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a5d5ad1 commit 7f88cc1

9 files changed

Lines changed: 397 additions & 24 deletions

File tree

.claude/CLAUDE.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,10 +82,10 @@ Registriert als MCP Server `aidex` (Prefix: `mcp__aidex__aidex_*`).
8282

8383
Status: `backlog → active → done | cancelled`
8484

85-
### Screenshots (v1.9+)
85+
### Screenshots (v1.9+, Optimierung v1.13)
8686
| Tool | Beschreibung |
8787
|------|--------------|
88-
| `aidex_screenshot` | Screenshot aufnehmen (fullscreen/active_window/window/region) |
88+
| `aidex_screenshot` | Screenshot aufnehmen + optional optimieren (`scale`, `colors`) |
8989
| `aidex_windows` | Offene Fenster auflisten (Helper für window-Modus) |
9090

9191
### Global Search (v1.11+)
@@ -193,20 +193,24 @@ aidex_tasks({ path: ".", status: "active", tag: "bug" }) # Gefiltert
193193
- Auto-Log bei Status-Änderungen und Task-Erstellung
194194
- Viewer: Tasks-Tab mit Priority-Farben, Done-Toggle, Cancelled-Sektion (durchgestrichen)
195195

196-
### Screenshots (v1.9)
196+
### Screenshots (v1.9, Optimierung v1.13)
197197
```
198198
aidex_screenshot() # Ganzer Bildschirm
199199
aidex_screenshot({ mode: "active_window" }) # Aktives Fenster
200200
aidex_screenshot({ mode: "window", window_title: "VS Code" }) # Bestimmtes Fenster
201+
aidex_screenshot({ scale: 0.5, colors: 2 }) # S/W, halbe Größe (ideal für LLM)
202+
aidex_screenshot({ colors: 16 }) # 16 Farben (UI lesbar)
201203
aidex_screenshot({ mode: "region" }) # Rechteck aufziehen
202-
aidex_screenshot({ delay: 3 }) # 3 Sek. warten
203204
aidex_windows({ filter: "chrome" }) # Fenster finden
204205
```
205206
- Kein Index nötig - standalone Tool
206207
- Cross-Platform: Windows (PowerShell), macOS (screencapture), Linux (maim/scrot)
207208
- Default: Speichert in `os.tmpdir()/aidex-screenshot.png` (überschreibt immer)
208209
- Optional: `filename` und `save_path` für andere Pfade
209210
- Rückgabe: Dateipfad → Claude kann sofort `Read` aufrufen
211+
- **LLM-Optimierung:** `scale` (0.1-1.0) und `colors` (2/4/16/256) reduzieren Dateigröße drastisch
212+
- **Strategie:** Start mit `scale: 0.5, colors: 2` → falls unlesbar `colors: 16` → dann `scale: 0.75`
213+
- Settings pro Fenster/App merken für die aktuelle Session
210214

211215
### Global Search (v1.11)
212216
```

README.md

Lines changed: 57 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ AiDex is an MCP server that gives AI coding assistants instant access to your en
2929
| **Global Search** | `global_init`, `global_query`, `global_signatures`, `global_status`, `global_refresh` | Search across ALL your projects at once — "Have I ever written X?" |
3030
| **Sessions** | `session`, `note` | Track sessions, detect external changes, leave notes for next session (with searchable history) |
3131
| **Task Backlog** | `task`, `tasks` | Built-in task management with priorities, tags, and auto-logged history |
32-
| **Screenshots** | `screenshot`, `windows` | Cross-platform screen capture (fullscreen, window, region)no index needed |
32+
| **Screenshots** | `screenshot`, `windows` | Cross-platform screen capture with LLM optimizationscale + color reduction saves up to 95% tokens |
3333
| **Viewer** | `viewer` | Interactive browser UI with file tree, signatures, tasks, and live reload |
3434

3535
**11 languages** — C#, TypeScript, JavaScript, Rust, Python, C, C++, Java, Go, PHP, Ruby
@@ -78,7 +78,7 @@ aidex_task({ path: ".", action: "create", title: "Fix edge case in parser", prio
7878
- [Session Notes](#session-notes)
7979
- [Task Backlog](#task-backlog)
8080
- [Global Search](#global-search)
81-
- [Screenshots](#screenshots)
81+
- [Screenshots — LLM-Optimized](#screenshots--llm-optimized)
8282
- [Interactive Viewer](#interactive-viewer)
8383
- [CLI Usage](#cli-usage)
8484
- [Performance](#performance)
@@ -309,7 +309,7 @@ Do I want to search code?
309309
| Global Search | `aidex_global_init`, `aidex_global_query`, `aidex_global_signatures`, `aidex_global_status`, `aidex_global_refresh` | Search across ALL projects |
310310
| Sessions | `aidex_session`, `aidex_note` | Track sessions, leave notes (with searchable history) |
311311
| Tasks | `aidex_task`, `aidex_tasks` | Built-in backlog with priorities, tags, auto-logged history |
312-
| Screenshots | `aidex_screenshot`, `aidex_windows` | Cross-platform screen capture (no index needed) |
312+
| Screenshots | `aidex_screenshot`, `aidex_windows` | Screen capture with LLM optimization (scale + color reduction, no index needed) |
313313
| Viewer | `aidex_viewer` | Interactive browser UI with file tree, signatures, tasks |
314314
315315
**11 languages:** C#, TypeScript, JavaScript, Rust, Python, C, C++, Java, Go, PHP, Ruby
@@ -354,9 +354,17 @@ aidex_global_status({ sort: "recent" }) # List al
354354
aidex_screenshot() # Full screen
355355
aidex_screenshot({ mode: "active_window" }) # Active window
356356
aidex_screenshot({ mode: "window", window_title: "VS Code" }) # Specific window
357+
aidex_screenshot({ scale: 0.5, colors: 2 }) # B&W, half size (ideal for LLM)
358+
aidex_screenshot({ colors: 16 }) # 16 colors (UI readable)
357359
aidex_windows({ filter: "chrome" }) # Find window titles
358360
```
359361
No index needed. Returns file path → use `Read` to view immediately.
362+
363+
**LLM optimization strategy:** Always start with aggressive settings, then retry if unreadable:
364+
1. First try: `scale: 0.5, colors: 2` (B&W, half size — smallest possible)
365+
2. If unreadable: retry with `colors: 16` (adds shading for UI elements)
366+
3. If still unclear: `scale: 0.75` or omit `colors` for full quality
367+
4. **Remember** what works for each window/app during the session — don't retry every time.
360368
```
361369

362370
### 4. Index your project
@@ -392,7 +400,7 @@ aidex_init({ path: "/path/to/your/project" })
392400
| `aidex_viewer` | Open interactive project tree in browser |
393401
| `aidex_task` | Create, read, update, delete tasks with priority and tags |
394402
| `aidex_tasks` | List and filter tasks by status, priority, or tag |
395-
| `aidex_screenshot` | Take a screenshot (fullscreen, window, region) |
403+
| `aidex_screenshot` | Take a screenshot (fullscreen, window, region) with optional scale + color reduction |
396404
| `aidex_windows` | List open windows for screenshot targeting |
397405
| `aidex_global_init` | Scan directory tree, register all indexed projects in global DB |
398406
| `aidex_global_status` | List all registered projects with stats |
@@ -521,31 +529,63 @@ aidex_global_status({ sort: "recent" }) # Most re
521529
aidex_global_refresh() # Update stats, remove stale
522530
```
523531

524-
## Screenshots
532+
## Screenshots — LLM-Optimized
533+
534+
Take screenshots and **reduce them up to 95%** for LLM context. A typical screenshot goes from ~100 KB to ~5 KB — that's thousands of tokens saved per image.
535+
536+
### Why this matters
537+
538+
| | Raw Screenshot | Optimized (scale=0.5, colors=2) |
539+
|---|---|---|
540+
| **File size** | ~100-500 KB | ~5-15 KB |
541+
| **Tokens consumed** | ~5,000-25,000 | ~250-750 |
542+
| **Text readable?** | Yes | Yes |
543+
| **Colors** | 16M (24-bit) | 2 (black & white) |
525544

526-
Take cross-platform screenshots directly from your AI assistant - no manual file paths needed:
545+
Most screenshots in AI context are for reading text — error messages, logs, UI labels. You don't need 16 million colors for that.
546+
547+
### Usage
527548

528549
```
529-
aidex_screenshot() # Full screen
530-
aidex_screenshot({ mode: "active_window" }) # Active window
550+
aidex_screenshot() # Full screen (full quality)
551+
aidex_screenshot({ mode: "active_window" }) # Active window
531552
aidex_screenshot({ mode: "window", window_title: "VS Code" }) # Specific window
532-
aidex_screenshot({ mode: "region" }) # Interactive selection
553+
aidex_screenshot({ scale: 0.5, colors: 2 }) # B&W, half size (best for text)
554+
aidex_screenshot({ scale: 0.5, colors: 16 }) # 16 colors (UI readable)
555+
aidex_screenshot({ colors: 256 }) # 256 colors (good quality)
556+
aidex_screenshot({ mode: "region" }) # Interactive selection
533557
aidex_screenshot({ mode: "rect", x: 100, y: 200, width: 800, height: 600 }) # Coordinates
534-
aidex_windows({ filter: "chrome" }) # Find window titles
558+
aidex_windows({ filter: "chrome" }) # Find window titles
535559
```
536560

537-
**Features:**
561+
### Optimization parameters
562+
563+
| Parameter | Values | Description |
564+
|-----------|--------|-------------|
565+
| `scale` | 0.1 - 1.0 | Scale factor (0.5 = half resolution). Most HiDPI screens are 2-3x anyway. |
566+
| `colors` | 2, 4, 16, 256 | Color reduction. 2 = black & white, ideal for text screenshots. |
567+
568+
### Recommended strategy for AI assistants
569+
570+
The tool description tells LLMs to optimize automatically:
571+
572+
1. **Start aggressive**: `scale: 0.5, colors: 2` (smallest possible)
573+
2. **If unreadable**: retry with `colors: 16` (adds shading for UI elements)
574+
3. **If still unclear**: try `scale: 0.75` or full color
575+
4. **Remember**: cache what works per window/app for the rest of the session
576+
577+
This way the AI learns the right settings per app without wasting tokens on oversized images.
578+
579+
### Features
580+
538581
- **5 capture modes**: Fullscreen, active window, specific window (by title), interactive region selection, coordinate-based rectangle
539-
- **Cross-platform**: Windows (PowerShell), macOS (screencapture), Linux (maim/scrot)
582+
- **Cross-platform**: Windows (PowerShell + System.Drawing), macOS (sips + ImageMagick), Linux (ImageMagick)
540583
- **Multi-monitor**: Select which monitor to capture
541584
- **Delay**: Wait N seconds before capturing (e.g., to open a menu first)
542-
- **Auto-path**: Default saves to temp directory with fixed filename - your AI reads it immediately
585+
- **Size reporting**: Shows original → optimized size and percentage saved
586+
- **Auto-path**: Default saves to temp directory with fixed filename
543587
- **No index required**: Works standalone, no `.aidex/` needed
544588

545-
Use `aidex_windows` to find the exact window title, then `aidex_screenshot` with `mode: "window"` to capture it.
546-
547-
**Future:** Screenshots will be storable in the AiDex database - attach them to tasks for bug documentation, capture before/after states for refactoring, or persist GUI evidence across sessions.
548-
549589
## Interactive Viewer
550590

551591
Explore your indexed project visually in the browser:

src/commands/index.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,5 @@ export { files, type FilesParams, type FilesResult, type ProjectFile } from './f
1414
export { note, getSessionNote, type NoteParams, type NoteResult } from './note.js';
1515
export { session, updateSessionHeartbeat, getSessionInfo, formatSessionTime, formatDuration, type SessionParams, type SessionResult, type SessionInfo, type ChangedFile } from './session.js';
1616
export { task, tasks, type TaskParams, type TaskResult, type TasksParams, type TasksResult, type TaskAction } from './task.js';
17-
export { screenshot, listWindows, type ScreenshotParams, type ScreenshotResult, type ScreenshotMode, type WindowsParams, type WindowsResult, type WindowInfo } from './screenshot/index.js';
17+
export { screenshot, listWindows, type ScreenshotParams, type ScreenshotResult, type ScreenshotMode, type ScreenshotColors, type WindowsParams, type WindowsResult, type WindowInfo } from './screenshot/index.js';
1818
export { globalInit, globalStatus, globalQuery, invalidateGlobalCache, globalSignatures, globalRefresh, type GlobalInitParams, type GlobalInitResult, type UnindexedRepo, type GlobalStatusParams, type GlobalStatusResult, type GlobalStatusProject, type GlobalQueryParams, type GlobalQueryResult, type GlobalQueryMode, type GlobalSignaturesParams, type GlobalSignaturesResult, type SignatureKind, type GlobalRefreshParams, type GlobalRefreshResult } from './global/index.js';

src/commands/screenshot/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ export type {
88
ScreenshotParams,
99
ScreenshotResult,
1010
ScreenshotMode,
11+
ScreenshotColors,
1112
WindowsParams,
1213
WindowsResult,
1314
WindowInfo,

0 commit comments

Comments
 (0)