Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions docs/feishu.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,14 +165,14 @@ The gateway downloads and forwards image and text file attachments to the AI age
| Feishu msg_type | Handling |
|-----------------|----------|
| `text` | Text extracted, forwarded as prompt |
| `image` | Image downloaded, resized (max 1200px), JPEG compressed, base64 encoded → `ContentBlock::Image` |
| `image` | Image downloaded, resized (max 1200px), JPEG compressed, stored to `~/.openab/media/inbound/<uuid>` → `ContentBlock::Image` |
| `file` | Text files only (`.txt`, `.py`, `.rs`, `.md`, `.json`, etc., max 512KB). Non-text files (`.pdf`, `.zip`, etc.) are silently ignored. |
| `audio` | Voice message downloaded (opus/ogg, max 25MB), base64 encoded, forwarded to core. If `[stt]` is enabled, core transcribes via Whisper API and injects `[Voice message transcript]: ...` into the prompt. If STT is disabled or fails, the message is silently skipped. |
| `audio` | Voice message downloaded (opus/ogg, max 25MB), stored to filesystem, forwarded to core. If `[stt]` is enabled, core transcribes via Whisper API and injects `[Voice message transcript]: ...` into the prompt. If STT is disabled or fails, the message is silently skipped. |
| `post` | Rich text: text nodes extracted as prompt, `img` nodes downloaded as image attachments. This is the format Feishu uses when @mention + paste image in a group. |

**Group chat limitation:** Feishu does not allow @mention and image upload in the same message. However, @mention + paste (Ctrl+V) an image works — Feishu sends this as a `post` message containing both the mention and the image. Direct image upload (via the attachment button) cannot include @mention, so the bot will not respond in groups.

**Processing pipeline:** Gateway downloads media using `GET /im/v1/messages/{message_id}/resources/{key}?type=image` with `tenant_access_token`, resizes to max 1200px, compresses to JPEG (quality 75), base64 encodes, and embeds in the `GatewayEvent.content.attachments` field. OAB core decodes attachments into `ContentBlock::Image` or `ContentBlock::Text` for the AI agent.
**Processing pipeline:** Gateway downloads media using `GET /im/v1/messages/{message_id}/resources/{key}?type=image` with `tenant_access_token`, resizes to max 1200px, compresses to JPEG (quality 75), and stores to `~/.openab/media/inbound/<uuid>`. The file path is passed in `GatewayEvent.content.attachments[].path`. OAB core reads the file directly from disk and converts to `ContentBlock::Image` or `ContentBlock::Text` for the AI agent.

## Streaming (Typewriter)

Expand Down
2 changes: 1 addition & 1 deletion docs/google-chat.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ working_dir = "/home/agent"
- Inline code, fenced code blocks: pass through unchanged
- Tables and other unsupported syntax pass through as-is
- **Streaming (edit_message)** — when OAB streaming is enabled, the bot edits its initial reply in-place as tokens arrive (typewriter effect)
- **Inbound attachments** — image, text file, and audio attachments are downloaded via Google Chat Media API and forwarded to the agent as base64 (PR #731 pattern):
- **Inbound attachments** — image, text file, and audio attachments are downloaded via Google Chat Media API and stored to `~/.openab/media/inbound/<uuid>` (colocate filesystem store):
- Images: resized to ≤1200px JPEG (q75); GIFs preserved. Max 10 MB.
- Text files: only known text extensions (`.txt`, `.md`, `.json`, `.py`, `.rs`, etc.). Max 512 KB.
- Audio: forwarded as-is for STT processing by core. Max 25 MB.
Expand Down
100 changes: 100 additions & 0 deletions docs/inbound-attachments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Inbound Attachments

How OAB handles images, audio, and files sent by users across all platforms.

## Architecture

```
User sends media (photo/voice/file)
→ Platform webhook delivers to Gateway
→ Gateway downloads via platform API (auth stays in Gateway)
→ Image: resize ≤1200px, JPEG compress (GIF passthrough ≤5MB)
→ Store to ~/.openab/media/inbound/<uuid>
→ WS event includes file path in attachments[].path
→ Core reads from disk (zero encoding overhead)
→ Processes: image → LLM, audio → STT, text_file → code block
→ File auto-evicted after 2 minutes
```

## Platform Support Matrix

| Platform | Images | Audio/Voice | Text Files | Video | Binary Files |
|----------|--------|-------------|------------|-------|--------------|
| **Discord** | ✅ | ✅ (STT) | ✅ | metadata only | skipped |
| **Telegram** | ✅ | ✅ (STT) | ✅ (whitelist) | skipped | skipped |
| **Feishu** | ✅ | ✅ (STT) | ✅ (whitelist) | skipped | skipped |
| **Google Chat** | ✅ | ✅ (STT) | ✅ (whitelist) | skipped | Drive files skipped |
| **WeCom** | ✅ | — | ✅ (whitelist) | skipped | skipped |
| **LINE** | planned | planned | — | — | — |
| **Slack** | ✅ | ✅ (STT) | ✅ | — | skipped |

## Processing Pipeline

### Images

1. Gateway downloads from platform API
2. `resize_and_compress()` — longest side ≤1200px, JPEG quality 75
3. GIFs ≤5MB passed through unchanged (preserves animation)
4. Stored to `~/.openab/media/inbound/<uuid>`
5. Core reads bytes → `ContentBlock::Image` → sent to LLM

### Audio / Voice Messages

1. Gateway downloads raw audio (ogg/m4a/mp3)
2. Stored to filesystem (no transcoding)
3. Core reads bytes → STT transcription (Whisper/Groq) → `[Voice message transcript]: ...`
4. If STT disabled: silently skipped

### Text Files (Documents)

1. Gateway downloads file
2. Extension whitelist check: `.txt`, `.csv`, `.md`, `.json`, `.yaml`, `.rs`, `.py`, `.js`, `.ts`, `.go`, `.java`, `.c`, `.cpp`, `.sh`, `.sql`, `.html`, `.css`, `.toml`, `.xml`, `.ini`, `.cfg`, `.conf`, etc.
3. UTF-8 validation — non-UTF-8 files rejected
4. Stored to filesystem
5. Core reads → wraps in markdown code block: `` ```filename.ext\n<content>\n``` ``

### Unsupported Types

Binary files (zip, pdf, exe, docx), video, and stickers are **silently skipped**. The agent does not receive any notification that a file was sent.

## Size Limits

| Type | Max Size | Enforced By |
|------|----------|-------------|
| Images | 10 MB | Gateway (pre-download Content-Length + post-download bytes) |
| Audio | 20 MB | Gateway |
| Text files | 20 MB | Gateway (same as store cap) |
| GIF passthrough | 5 MB | `resize_and_compress()` |
| Store (defense-in-depth) | 20 MB | `store_media()` |

## Storage (Colocate Mode)

Media is stored at `~/.openab/media/inbound/<uuid>`:

- **Filenames**: Server-generated UUID v4, no extension (MIME type in event payload)
- **TTL**: 2 minutes — background task evicts expired files every 30 seconds
- **Trust boundary**: Gateway and Core share the same `$HOME` (same pod / sidecar)
- **No auth required**: Core reads directly from filesystem, no HTTP/token needed

### Security

- **Path traversal**: Impossible — filenames are UUID only, never user-supplied
- **Token leakage**: Platform auth tokens (Telegram bot token, LINE access token, Feishu tenant token) stay in Gateway, never reach Core or agent
- **Disk exhaustion**: TTL eviction + size limits prevent unbounded growth
- **No executable content**: Files are raw data, never executed

### Future: HTTP Proxy Mode

For separated deployments (Gateway ≠ Core pod), a future PR will add `GET /media/<uuid>` on the Gateway, allowing Core to fetch via internal HTTP. The `attachments[].path` field will be replaced by `attachments[].url` in that mode.

## Configuration

No additional configuration required. The filesystem store is always active when Gateway is running. Ensure Gateway and Core share the same `$HOME` (default in Helm colocate/sidecar mode).

## Related

- [Telegram](telegram.md) — Telegram-specific behavior and limitations
- [Feishu](feishu.md) — Feishu image/file/audio handling
- [Google Chat](google-chat.md) — Google Chat attachment support
- [STT (Speech-to-Text)](stt.md) — Audio transcription configuration
- [Sending Files (Outbound)](sendfiles.md) — Agent → user file delivery (separate mechanism)
13 changes: 13 additions & 0 deletions docs/telegram.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,19 @@ explain VPC peering ← ignored in groups

DMs and replies within forum topics always trigger the agent (no @mention needed).

### File Attachments (Inbound)

The gateway downloads media from Telegram and stores it locally (`~/.openab/media/inbound/<uuid>`). Core reads directly from disk — no base64 encoding overhead.

| Type | Handling |
|------|----------|
| **Images** | Downloaded, resized (max 1200px), JPEG compressed, stored to filesystem. Agent sees the image. |
| **Documents** | Text-based files (`.txt`, `.csv`, `.rs`, `.py`, etc.) up to 20MB read as UTF-8 and passed to agent. Binary files silently skipped. |
| **Audio/Voice** | Downloaded and stored. If STT is enabled in Core, automatically transcribed and passed as text. |

**Not supported (inbound):** video, stickers, animations (silently skipped).
**Not supported (outbound):** bot cannot send images/files back to the user yet.

### Emoji reactions

The bot shows status reactions on your message as the agent works:
Expand Down
19 changes: 9 additions & 10 deletions gateway/src/adapters/feishu.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1449,15 +1449,15 @@ pub async fn download_feishu_image(
return None;
}
};
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(&compressed);
let path = crate::store::store_media(&compressed).await?;
let ext = if mime == "image/gif" { "gif" } else { "jpg" };
Some(crate::schema::Attachment {
attachment_type: "image".into(),
filename: format!("{}.{}", image_key, ext),
mime_type: mime,
data,
data: String::new(),
size: compressed.len() as u64,
path: Some(path),
})
}

Expand Down Expand Up @@ -1511,15 +1511,14 @@ pub async fn download_feishu_file(
tracing::warn!(file_name, size = bytes.len(), "feishu file exceeds 512KB limit");
return None;
}
let text = String::from_utf8_lossy(&bytes);
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(text.as_bytes());
let path = crate::store::store_media(&bytes).await?;
Some(crate::schema::Attachment {
attachment_type: "text_file".into(),
filename: file_name.to_string(),
mime_type: "text/plain".into(),
data,
data: String::new(),
size: bytes.len() as u64,
path: Some(path),
})
}

Expand Down Expand Up @@ -1569,14 +1568,14 @@ pub async fn download_feishu_audio(
return None;
}
tracing::debug!(file_key, size = bytes.len(), "feishu audio downloaded");
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(&bytes);
let path = crate::store::store_media(&bytes).await?;
Some(crate::schema::Attachment {
attachment_type: "audio".into(),
filename: format!("{}.ogg", file_key),
mime_type: content_type,
data,
data: String::new(),
size: bytes.len() as u64,
path: Some(path),
})
}

Expand Down
22 changes: 11 additions & 11 deletions gateway/src/adapters/googlechat.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1257,14 +1257,14 @@ pub async fn download_googlechat_image(
return None;
}
};
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(&compressed);
let path = crate::store::store_media(&compressed).await?;
Some(crate::schema::Attachment {
attachment_type: "image".into(),
filename: content_name.to_string(),
mime_type: mime,
data,
data: String::new(),
size: compressed.len() as u64,
path: Some(path),
})
}

Expand Down Expand Up @@ -1309,18 +1309,18 @@ pub async fn download_googlechat_file(
warn!(content_name, size = bytes.len(), limit = max_size, "googlechat file exceeds size limit");
return None;
}
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(&bytes);
let path = crate::store::store_media(&bytes).await?;
Some(crate::schema::Attachment {
attachment_type: "text_file".into(),
filename: content_name.to_string(),
mime_type: "text/plain".into(),
data,
data: String::new(),
size: bytes.len() as u64,
path: Some(path),
})
}

/// Download an audio attachment as-is (no resize/transcode) → base64.
/// Download an audio attachment as-is (no resize/transcode) → filesystem store.
/// Core's STT pipeline (when available) consumes this as `audio` attachment_type.
pub async fn download_googlechat_audio(
client: &reqwest::Client,
Expand Down Expand Up @@ -1355,14 +1355,14 @@ pub async fn download_googlechat_audio(
warn!(content_name, size = bytes.len(), "googlechat audio exceeds 25MB limit");
return None;
}
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(&bytes);
let path = crate::store::store_media(&bytes).await?;
Some(crate::schema::Attachment {
attachment_type: "audio".into(),
filename: content_name.to_string(),
mime_type: content_type.to_string(),
data,
data: String::new(),
size: bytes.len() as u64,
path: Some(path),
})
}

Expand Down Expand Up @@ -2315,7 +2315,7 @@ mod tests {
assert_eq!(att.attachment_type, "image");
assert_eq!(att.filename, "photo.png");
assert_eq!(att.mime_type, "image/jpeg"); // resized PNG → JPEG
assert!(!att.data.is_empty());
assert!(att.path.is_some()); // stored to filesystem
assert!(att.size > 0);
}

Expand Down
Loading
Loading