Skip to content

Commit c35b44e

Browse files
chaodu-agent超渡法師
chaodu-agent
and
超渡法師
authored
docs(adr): iMessage integration architecture (#1141)
* docs(adr): iMessage integration architecture Proposes two integration paths for iMessage as an OAB channel: 1. Self-hosted — Mac mini running imessage-bridge (poll chat.db + AppleScript) 2. Cloud — Photon Spectrum SDK (managed iMessage infrastructure) Both implement ChatAdapter trait and connect via Custom Gateway. Covers architecture, message flow, security analysis, and phased plan. * docs(adr): add @mention parsing details and group chat limitations - Add §6.1 explaining how mentions are stored in attributedBody blob - Compare mention detection across platforms (Discord/LINE/Slack/iMessage) - Clarify implications for Phase 1 (1:1) vs Phase 5 (group) - Update open question #3 with recommendation to defer group chat - Add @mention parsing to Phase 5 scope * docs(adr): rewrite architecture — gateway on macOS host, OAB in OrbStack Major changes: - Primary architecture: iMessage adapter built into Custom Gateway running natively on macOS, not as a separate bridge binary - Gateway connects to OAB core in OrbStack via WebSocket (outbound) - Removed split bridge/gateway design — single binary on host - Added explanation why containerization is not possible (TCC, WAL, AppleScript IPC) - Simplified phases: 3 phases instead of 5 - Removed bridge language open question (gateway is Rust) - Kept Photon Spectrum as alternative path --------- Co-authored-by: 超渡法師 <chaodu-agent@openab.dev>
1 parent 5c1251a commit c35b44e

1 file changed

Lines changed: 366 additions & 0 deletions

File tree

‎docs/adr/imessage-integration.md‎

Lines changed: 366 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,366 @@
1+
# ADR: iMessage Integration via macOS Gateway
2+
3+
- **Status:** Proposed
4+
- **Date:** 2026-06-18
5+
- **Author:** @chaodu-agent
6+
- **Reviewers:** @pahud
7+
- **Tracking issues:** (none yet)
8+
9+
---
10+
11+
## 1. Context & Decision
12+
13+
Enable OAB agents to receive and respond to iMessage conversations, allowing users to interact with their agent team through Apple's native messaging platform. This extends OAB's multi-platform adapter architecture (see [ADR: Multi-Platform Adapters](./multi-platform-adapters.md)) to a platform that lacks an official API.
14+
15+
**Decision:** Implement iMessage as a built-in adapter within the Custom Gateway, running natively on macOS. The gateway polls `chat.db` for inbound messages and sends outbound via AppleScript, connecting to OAB core (running in OrbStack/containers) over WebSocket. This is an iMessage-only deployment — no separate bridge binary needed.
16+
17+
---
18+
19+
## 2. Motivation
20+
21+
- Users want to interact with OAB agents from iMessage — the default messaging app on iOS/macOS with 1B+ active users
22+
- iMessage offers a more personal, low-friction interaction surface compared to Discord/Slack
23+
- Apple provides **no official iMessage API** — all third-party integrations rely on macOS Messages.app as a bridge
24+
- The existing `agy-acp` component already proves the "poll SQLite DB" pattern works reliably in OAB
25+
- iMessage bridging requires macOS-native access (`chat.db` + AppleScript) — it **cannot** run inside a container
26+
27+
### Why iMessage Over Existing Channels
28+
29+
iMessage fills a gap that LINE, Telegram, and Slack cannot:
30+
31+
| Advantage | Detail |
32+
|-----------|--------|
33+
| **North America default** | iPhone-to-iPhone messaging uses iMessage automatically — no app install required. Dominant in US/Canada market. |
34+
| **Zero-friction reach** | Only requires a phone number. No "add friend" / "find bot username" step. Ideal for cold outreach: customer support, appointment reminders, order notifications. |
35+
| **High trust signal** | Conversations appear alongside friends/family in the native Messages.app. Users perceive it as personal communication, not "yet another bot." |
36+
| **Apple ecosystem integration** | Siri dictation, Apple Watch, CarPlay, Focus Mode — notifications are not filtered as "app push." |
37+
| **No new app required** | Enterprise scenario: clients/employees already have iPhones; no need to mandate LINE/Telegram/Slack installation. |
38+
39+
**When NOT to use iMessage:**
40+
41+
| Scenario | Better choice |
42+
|----------|--------------|
43+
| Asia market (Taiwan/Japan/Thailand) | LINE |
44+
| Developer/tech communities | Discord / Telegram |
45+
| Cross-platform users (Android + iPhone) | Telegram / WhatsApp |
46+
| Rich UI (buttons, carousels) | LINE / Slack |
47+
| Group bot interactions | Discord / Slack |
48+
| Message editing / threading | Discord / Slack |
49+
50+
**Summary:** iMessage's core value is **North American market + zero-install barrier + high trust perception**. For Asian markets or technical communities, LINE/Telegram remain more practical.
51+
52+
---
53+
54+
## 3. Architecture
55+
56+
### 3.1 Primary: Custom Gateway on macOS Host + OAB in OrbStack
57+
58+
```
59+
┌────────────────┐
60+
│ iPhone User │
61+
│ (iMessage) │
62+
└───────┬────────┘
63+
│
64+
▼
65+
═══════════════════
66+
║ Apple iMessage ║
67+
║ Network ║
68+
═══════════════════
69+
│
70+
▼
71+
┌──────────────────────────────────────────┐
72+
│ macOS host (Mac mini / MacBook) │
73+
│ │
74+
│ ┌────────────────────────────────────┐ │
75+
│ │ Messages.app │ │
76+
│ │ ~/Library/Messages/chat.db │ │
77+
│ └──────────────┬─────────────────────┘ │
78+
│ │ poll every 100-500ms │
79+
│ ┌──────────────▼─────────────────────┐ │
80+
│ │ Custom Gateway (native binary) │ │
81+
│ │ │ │
82+
│ │ ├── iMessage adapter (built-in) │ │
83+
│ │ │ ├── poll chat.db (SQLite) │ │
84+
│ │ │ └── send via osascript │ │
85+
│ │ └── WebSocket client → OAB │ │
86+
│ └──────────────┬─────────────────────┘ │
87+
│ │ │
88+
└─────────────────┼─────────────────────────┘
89+
│ WebSocket (outbound connection)
90+
▼
91+
┌──────────────────────────────────────────┐
92+
│ OrbStack (Linux containers) │
93+
│ │
94+
│ ┌────────────────────────────────────┐ │
95+
│ │ OAB Core │ │
96+
│ │ ├── AdapterRouter │ │
97+
│ │ ├── SessionPool │ │
98+
│ │ └── ACP agents │ │
99+
│ └────────────────────────────────────┘ │
100+
│ │
101+
└──────────────────────────────────────────┘
102+
```
103+
104+
**Why the gateway must run on macOS host (not in a container):**
105+
106+
1. **`chat.db` access** — macOS TCC (Transparency, Consent, Control) grants Full Disk Access per-app. A containerized process cannot inherit FDA permissions.
107+
2. **SQLite WAL locking** — `chat.db` is held by Messages.app with WAL lock. A cross-VM mount risks lock contention.
108+
3. **AppleScript IPC** — `osascript` communicates with Messages.app via macOS IPC (Apple Events). This only works from the same macOS session.
109+
110+
### 3.2 Alternative: Photon Spectrum (Cloud, No Mac Required)
111+
112+
```
113+
┌────────────────┐ ═══════════════ ┌──────────────────┐
114+
│ iPhone User │───────║ Apple iMessage ║──────│ Photon Cloud │
115+
│ (iMessage) │ ═══════════════ │ (managed Mac群) │
116+
└────────────────┘ └────────┬─────────┘
117+
│ gRPC stream
118+
▼
119+
┌─────────────────────────────────────────────────────────────────────┐
120+
│ OrbStack (Linux containers) │
121+
│ │
122+
│ ┌──────────────────┐ ┌───────────────────────────────────────┐ │
123+
│ │ Spectrum Sidecar │────►│ OAB Core │ │
124+
│ │ (Node.js/Bun) │ │ │ │
125+
│ └──────────────────┘ └───────────────────────────────────────┘ │
126+
└─────────────────────────────────────────────────────────────────────┘
127+
```
128+
129+
Photon Spectrum manages the Mac infrastructure — no local Mac needed. Useful as fallback if Apple restricts local `chat.db` access.
130+
131+
---
132+
133+
## 4. How iMessage Bridging Works
134+
135+
Apple does not provide an iMessage API. The gateway's iMessage adapter relies on two macOS-native mechanisms:
136+
137+
### 4.1 Receiving Messages (Inbound)
138+
139+
Messages.app writes all received messages to a local SQLite database at `~/Library/Messages/chat.db`. The adapter polls this DB for new rows:
140+
141+
```sql
142+
SELECT rowid, text, handle_id, date, is_from_me, cache_roomnames
143+
FROM message
144+
WHERE rowid > ?last_seen_rowid
145+
ORDER BY rowid ASC
146+
```
147+
148+
This is the **same pattern as `agy-acp`**, which polls `conversations/*.db` for new `step_payload` rows. Both use:
149+
- SQLite read-only connection with WAL mode
150+
- Polling interval (100-500ms)
151+
- Monotonically increasing row ID as cursor
152+
153+
### 4.2 Sending Messages (Outbound)
154+
155+
Messages are sent by invoking AppleScript via `osascript`:
156+
157+
```applescript
158+
tell application "Messages"
159+
set targetService to 1st account whose service type = iMessage
160+
set targetBuddy to participant "+1234567890" of targetService
161+
send "Hello from OAB" to targetBuddy
162+
end tell
163+
```
164+
165+
For group chats, send to a specific chat ID:
166+
167+
```applescript
168+
tell application "Messages"
169+
set targetChat to chat id "iMessage;+;chat123456"
170+
send "Hello from OAB" to targetChat
171+
end tell
172+
```
173+
174+
### 4.3 Comparison with agy-acp
175+
176+
| Aspect | agy-acp | Custom Gateway (iMessage adapter) |
177+
|--------|---------|-----------------------------------|
178+
| Monitored program | `agy` (Gemini CLI) | Messages.app |
179+
| Data source | `conversations/*.db` | `~/Library/Messages/chat.db` |
180+
| Poll mechanism | `WHERE idx > last` every 100ms | `WHERE rowid > last` every 100-500ms |
181+
| Data format | protobuf `step_payload` field 20.1 | `attributedBody` blob / plain `text` column |
182+
| Send mechanism | spawn `agy -p "prompt"` | spawn `osascript` (AppleScript) |
183+
| Output | JSON-RPC streaming notifications | WebSocket to OAB core (direct) |
184+
185+
---
186+
187+
## 5. Message Flow
188+
189+
### Inbound (User → Agent)
190+
191+
```
192+
1. User sends iMessage from iPhone
193+
2. Apple iMessage network delivers to Mac's Messages.app
194+
3. Messages.app writes row to chat.db
195+
4. Gateway's iMessage adapter detects new row (poll)
196+
5. Adapter formats as OpenAB inbound event:
197+
{ "platform": "imessage", "sender": "+1234567890",
198+
"text": "...", "channel_id": "iMessage;-;+1234567890" }
199+
6. Gateway sends event to OAB core via WebSocket
200+
7. AdapterRouter dispatches to SessionPool → agent
201+
```
202+
203+
### Outbound (Agent → User)
204+
205+
```
206+
1. Agent produces response via ACP session
207+
2. OAB core sends outbound event via WebSocket to gateway
208+
3. Gateway's iMessage adapter invokes AppleScript
209+
4. Messages.app → Apple network → User's iPhone
210+
```
211+
212+
---
213+
214+
## 6. Platform Limitations
215+
216+
- No message editing (iMessage supports "edit" natively on iOS 16+ but AppleScript cannot trigger it)
217+
- No threading (conversations are flat)
218+
- Reactions map to tapbacks (❤️, 👍, 👎, 😂, ‼️, ❓) — only 6 options
219+
- No typing indicators via AppleScript (Spectrum Cloud supports this)
220+
- No structured @mention field (see §6.1 below)
221+
222+
### 6.1 @Mention Detection (Group Chat)
223+
224+
iMessage supports @mentions (iOS 14+, displayed as bold blue text), but `chat.db` does **not** expose them as a structured column. The mention data is embedded inside the `attributedBody` blob — a serialized `NSAttributedString` (NSKeyedArchiver / typedstream format).
225+
226+
**To extract mentions, the adapter must:**
227+
228+
1. Read `message.attributedBody` (binary blob)
229+
2. Decode NSKeyedArchiver binary plist
230+
3. Locate ranges where `__kIMMessagePartAttributeName` = 1 (indicates a mention)
231+
4. Extract the mentioned handle ID from `__kIMMentionConfirmedMention`
232+
233+
**Comparison with other platforms:**
234+
235+
| Platform | Mention detection | Complexity |
236+
|----------|-------------------|-----------|
237+
| Discord | `message.mentions` array | Trivial — structured field |
238+
| LINE | `mentionees` in webhook payload | Trivial — structured field |
239+
| Slack | `<@BOT_ID>` in text + `app_mention` event type | Easy — text pattern |
240+
| iMessage | Parse binary `attributedBody` blob | Hard — undocumented binary format |
241+
242+
**Implications for group chat:**
243+
- **1:1 conversations (Phase 1):** No mention detection needed — all messages are directed at the bot
244+
- **Group chat (Phase 3):** Adapter must parse `attributedBody` to know when the bot is mentioned, or fall back to keyword-prefix trigger (e.g. `/ask ...`)
245+
- The `attributedBody` format is undocumented and may change across macOS versions — Rust `plist` crate can decode the binary plist, but the internal schema requires reverse-engineering
246+
- **Outbound:** AppleScript `send` does not support sending @mentions — bot replies are plain text only
247+
248+
---
249+
250+
## 7. Config Design
251+
252+
```toml
253+
[gateway.imessage]
254+
enabled = true
255+
poll_interval_ms = 200
256+
chat_db_path = "~/Library/Messages/chat.db" # default; override for testing
257+
258+
# OAB core WebSocket endpoint (OrbStack container)
259+
[gateway]
260+
oab_ws_url = "ws://localhost:8080/ws" # OrbStack port-forwards to OAB
261+
262+
# Spectrum mode (alternative — no local Mac needed)
263+
[gateway.imessage.spectrum]
264+
enabled = false
265+
project_id = "${PHOTON_PROJECT_ID}"
266+
project_secret = "${PHOTON_PROJECT_SECRET}"
267+
```
268+
269+
---
270+
271+
## 8. Security Considerations
272+
273+
### 8.1 macOS Host Risks
274+
275+
| Risk | Mitigation |
276+
|------|-----------|
277+
| No container isolation | Dedicated macOS user account with minimal privileges |
278+
| Mac compromise → iMessage access | Gateway is a thin adapter; all agent logic stays in OrbStack |
279+
| Apple account credential exposure | Use a dedicated Apple ID, not personal |
280+
| AppleScript injection | Sanitize all outbound text; no user input in script template |
281+
| Full Disk Access requirement | Grant FDA only to the gateway binary, not the user shell |
282+
283+
### 8.2 Photon Spectrum Risks
284+
285+
| Risk | Mitigation |
286+
|------|-----------|
287+
| Third-party dependency | Photon manages Apple infra; you trust their SLA |
288+
| Shared phone numbers (free tier) | Upgrade to dedicated line ($250/mo) for consistent identity |
289+
| Apple ToS enforcement | Photon assumes this risk; self-hosted is fallback |
290+
| gRPC stream reliability | SDK handles auto-reconnect; SMS/RCS fallback |
291+
292+
### 8.3 General
293+
294+
- Messages contain PII — gateway and OAB must encrypt in transit (WebSocket over TLS)
295+
- Rate limiting on outbound to avoid Apple throttling/blocking
296+
- Gateway binary should be code-signed to satisfy macOS Gatekeeper
297+
298+
---
299+
300+
## 9. Deployment
301+
302+
### Primary: macOS + OrbStack
303+
304+
```
305+
macOS host:
306+
- Custom Gateway binary (Rust, native arm64/x86_64)
307+
- Runs as launchd service (auto-restart, auto-start on boot)
308+
- Requires: Full Disk Access for chat.db, Accessibility for AppleScript
309+
310+
OrbStack (on same Mac):
311+
- OAB Core container (Linux)
312+
- Exposes WebSocket endpoint on localhost
313+
```
314+
315+
**Why not containerize the gateway:**
316+
- `chat.db` cannot be volume-mounted into Linux VMs (macOS TCC blocks cross-process FDA)
317+
- AppleScript IPC (`osascript`) requires native macOS session — unavailable in containers
318+
- SQLite WAL mode may conflict with cross-VM file locking
319+
320+
| Deployment | Where | Cost | Complexity |
321+
|-----------|-------|------|------------|
322+
| Primary (this ADR) | Mac (gateway) + OrbStack (OAB) | ~$600 Mac mini | Low |
323+
| Spectrum alternative | OrbStack only (no Mac needed) | $0-250/mo | Low |
324+
325+
---
326+
327+
## 10. Implementation Phases
328+
329+
| Phase | Scope | Dependencies |
330+
|-------|-------|-------------|
331+
| **Phase 1** | iMessage adapter in Custom Gateway: poll chat.db, send via AppleScript, WebSocket to OAB | Custom Gateway ([ADR](./custom-gateway.md)) |
332+
| **Phase 2** | Spectrum sidecar adapter as alternative (no Mac required) | Photon account |
333+
| **Phase 3** | Rich features: tapback reactions, group chat support, @mention parsing, attachment handling | Phase 1 |
334+
335+
---
336+
337+
## 11. Apple Compliance & Risks
338+
339+
Apple does not provide an official iMessage API. All known approaches rely on:
340+
341+
1. **macOS Messages.app + SQLite** — reading `chat.db` (requires Full Disk Access)
342+
2. **AppleScript automation** — sending via `osascript` (uses public macOS APIs)
343+
3. **No protocol reverse-engineering** — no private framework usage
344+
345+
**Current landscape (as of 2026-06):**
346+
- BlueBubbles, AirMessage have operated for years without Apple enforcement
347+
- Photon Spectrum launched April 2026, commercially offering managed iMessage lines
348+
- Apple has not issued cease-and-desist to any known project
349+
- Risk: Apple could restrict `chat.db` access or AppleScript Messages automation in a future macOS update
350+
351+
**Mitigation:** The adapter architecture is modular — if Apple blocks the self-hosted path, Spectrum Cloud remains as fallback (they absorb the compliance risk). If both paths are blocked, the adapter can be disabled without affecting other OAB platforms.
352+
353+
---
354+
355+
## 12. Open Questions
356+
357+
| # | Question | Options | Notes |
358+
|---|----------|---------|-------|
359+
| 1 | Poll interval default | 100ms vs 200ms vs 500ms | Tradeoff: latency vs CPU. agy-acp uses 100ms |
360+
| 2 | Group chat support in Phase 1? | Yes / defer to Phase 3 | Recommend defer — no structured @mention field means bot can't reliably detect when addressed. 1:1 is the sweet spot. |
361+
| 3 | Should gateway run as launchd service? | Yes (auto-restart) / manual | launchd is macOS best practice for daemons |
362+
| 4 | Photon free tier shared numbers acceptable? | Yes for POC / require dedicated | Shared numbers may confuse recipients |
363+
364+
---
365+
366+
_This ADR was drafted based on research into Photon Spectrum (photon-hq/spectrum-ts), imessage-kit, and the existing agy-acp polling pattern in OAB._

0 commit comments

Comments
 (0)