tracking(serve): Bound multi-workspace daemon resource usage

[doudouOUC]

What would you like to be added?

Track and deliver bounded resource usage for the production qwen serve multi-workspace daemon.

The daemon currently limits registered workspaces and per-workspace sessions, but count-only limits do not bound bytes held by request bodies, WebSocket assembly, outbound delivery, replay buffers, session restore, background work, child-process output, or generation-scoped caches.

The work is split into three phases:

1. Phase 1 adds a daemon-owned, non-waiting resource budget; bounded HTTP, WebSocket, NDJSON, event, replay, session restore, export, and process-output paths; fair heavy/spawn/process scheduling; generation-scoped workspace resource disposal; observe-only process-tree/cgroup memory status; and writer-safe, process-tree-aware shutdown.
2. Phase 2 may add explicitly enabled aggregate RSS/cgroup admission and enforcement.
3. Phase 3 may add opt-in idle session or runtime reclamation.

Phase 1 acceptance criteria:

• Every production qwen serve root and managed restore path has an explicit resource owner, admission point, and release point.
• Daemon-owned variable buffers, queues, caches, readers, and buffered child-process output have both count and byte limits.
• Dynamic workspace removal and failed initialization release generation-scoped resources without allowing late asynchronous work to recreate them.
• Oversized work fails with stable REST and ACP errors while the daemon remains responsive and other workspace generations remain isolated.
• Memory observation reports process-tree and cgroup signals without triggering GC, LRU, session close, channel reclamation, or process termination.
• Shutdown drains or aborts writers before terminating root-owned process groups and reports an unclean non-zero exit when resources cannot be verified as released.

Explicit non-goals for Phase 1 are workspace/session capacity LRU, active-session eviction, RSS-triggered destructive remediation, and a generic exactly-once operation receipt service.

Why is this needed?

A maximum workspace or session count is not a memory-safety boundary. One abnormal transcript, slow client, oversized response, unbounded child-process output, or repeated workspace register/remove churn can still exhaust the daemon and affect unrelated workspaces. Phase 1 establishes deterministic admission and failure semantics before any destructive pressure policy is considered.

Additional context

• RFC: Support multiple workspaces in one qwen serve daemon #6378 established the multi-workspace baseline and recommends focused hardening issues for follow-up work.
• fix(serve): Isolate daemon session maintenance writers #7975 provides writer-safe session maintenance that the shutdown work should build on or rebase onto.
• feat(serve): establish workspace runtime ownership #7308 is related to runtime ownership but is not a Phase 1 dependency.
• Process containment references: Node.js child processes, Linux cgroup v2, and Windows taskkill.

Status

Delivery tracker: #8091. Phase 1 is not complete. #8091 is the authoritative workstream checklist; this section is a concise snapshot as of 2026-08-19.

Landed

• feat(serve): resolve and report the daemon memory budget #8245 — resolves and reports the configured and effective daemon memory budget, host/cgroup source, modeled root reserve and child pool, with enforced: false.
• feat(serve): observe daemon and child memory against real denominators #8423 and feat(serve): report aggregate ACP child RSS, not just the primary's #8462 — observe root memory pressure and aggregate self-reported RSS for active ACP children without remediation. Coverage remains intentionally partial rather than process-tree complete.
• refactor(serve): model a per-child heap partition of the daemon budget #8508 — models a fixed per-child heap partition and maximum concurrent-child count. The model remains off | observe; no child heap argument or admission policy is applied.
• feat(serve): measure ACP child peak old-generation heap #9380 — measures lifetime whole-old-generation committed and post-major-GC live-set peaks inside each daemon ACP child, together with peak total heap, major-GC count/pause time, and unclassified-space reporting. It remains observation-only and does not change child spawn arguments or admission.
• fix(cli): Bound ACP textual tool-result payloads #8450 — bounds canonical textual ACP tool-result display fields while preserving canonical transcripts and offline exports.
• feat(serve): bound daemon ACP NDJSON buffers #8911 and fix(serve): Close daemon ACP resource guard gaps #8947 — bound daemon-owned ACP NDJSON frames, decoded queues, SDK handler/outbound queues, and transport failure cleanup for the exact workspace channel generation.
• fix(serve): Bound ACP HTTP pre-attach buffers by bytes #9007 — bounds ACP HTTP pre-attach operations by count and retained bytes, owns accounting through delivery settlement, and rolls back provisional session ownership when delivery fails.

In flight

• bug(serve): daemon authorises each ACP child 50% of host memory, never divided by child count #8182 — tracks the remaining child-capacity behavioral policy. Peak whole-old-generation measurement landed in feat(serve): measure ACP child peak old-generation heap #9380; enforcement now awaits representative workload comparison against the modeled ceiling plus race-safe active and in-flight spawn accounting.

Remaining Phase 1 work

• Complete process-tree observation for hung children, channel workers, MCP descendants, and separate cgroup current usage.
• Finish the remaining HTTP, WebSocket, SSE, child-RPC, IPC, and other daemon-owned transport/container bounds.
• Bound event, replay, prompt, persistence, session restore/export, workspace-supplied inputs, and buffered external-process output.
• Add narrow per-workspace and process-wide retained-byte quotas at actual container insertion/removal points.
• Close the remaining generation-scoped initialization, replacement, removal, cache, and background-work lifecycle gaps.
• Finish writer-safe, process-tree-aware shutdown and unclean-exit reporting.

Phase 2 aggregate RSS/cgroup enforcement and Phase 3 opt-in idle session/runtime reclamation remain future optional work and have not started.

[qwen-code-ci-bot]

✅ Qwen Triage finished — view run. See the stage comments in this thread for the result.

✅ Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

[qwen-code-ci-bot]

Triage

• Type: feature request (tracking issue)
• Labels: type/feature-request, category/core, category/performance, scope/session-management, scope/memory-usage, priority/P2, need-discussion, daemon
• Next: accept for exploration — phased resource bounding for the production daemon

---

Well-structured tracking issue. The daemon (qwen serve) has seen steady investment (workspace Channel API, session writer isolation, trust hot-reload, managed shutdown) and count-only limits are a known gap for multi-workspace production use. The phased approach — admission/failure semantics first, enforcement second, reclamation third — is the right order: you need deterministic failure before you can safely enforce.

A few observations:

• Phase 1 scope is large. Six bounded paths (HTTP, WebSocket, NDJSON, event, replay, session restore) plus scheduling, disposal, observation, and shutdown is a lot for one phase. Consider whether the acceptance criteria can be met incrementally — e.g., HTTP + WebSocket + shutdown first, then the remaining paths — so each PR stays reviewable.
• Relationship to fix(serve): Isolate daemon session maintenance writers #7975 and feat(serve): establish workspace runtime ownership #7308 is clear. fix(serve): Isolate daemon session maintenance writers #7975 (writer-safe session maintenance) is a prerequisite for the shutdown work; feat(serve): establish workspace runtime ownership #7308 (runtime ownership) is related but not blocking. Good that these are called out explicitly.
• Non-goals are well-defined. Excluding LRU eviction, active-session eviction, and RSS-triggered remediation from Phase 1 keeps the scope honest.

CHANGELOG confirms active daemon development (feat(serve): expose workspace Channel management API, fix(serve): Release managed session writer locks on shutdown, fix(daemon): harden Todo Stop Guard continuations, among others). Resource bounding is a natural next step.

Verdict: accept for exploration. The need-discussion label is appropriate — this tracking issue should drive focused implementation PRs, each small enough to review and revert independently.

中文说明

• 类型: 功能请求（跟踪 issue）
• 标签: type/feature-request、category/core、category/performance、scope/session-management、scope/memory-usage、priority/P2、need-discussion、daemon
• 下一步: 接受探索——为生产守护进程分阶段设定资源边界

---

结构清晰的跟踪 issue。守护进程（qwen serve）持续投入中（workspace Channel API、session writer 隔离、trust 热重载、托管关闭等），仅靠数量限制无法覆盖多 workspace 生产场景。分阶段方案——先确定准入/失败语义，再强制执行，最后回收——顺序正确：需要先有确定性失败，才能安全地执行限制。

几点观察：

• Phase 1 范围较大。 六条有界路径（HTTP、WebSocket、NDJSON、event、replay、session restore）加上调度、释放、观测和关闭，作为一个阶段内容偏多。建议考虑增量交付——例如先做 HTTP + WebSocket + shutdown，再做其余路径——让每个 PR 保持可审查。
• 与 fix(serve): Isolate daemon session maintenance writers #7975 和 feat(serve): establish workspace runtime ownership #7308 的关系清晰。 fix(serve): Isolate daemon session maintenance writers #7975（writer-safe session 维护）是关闭工作的前置；feat(serve): establish workspace runtime ownership #7308（运行时所有权）相关但不阻塞。明确标注了这些依赖，很好。
• 非目标定义明确。 Phase 1 排除 LRU 驱逐、活跃 session 驱逐和 RSS 触发的修复措施，范围诚实。

CHANGELOG 确认守护进程活跃开发中。资源边界设定是自然的下一步。

结论: 接受探索。need-discussion 标签合适——此跟踪 issue 应驱动聚焦的实现 PR，每个 PR 小到可以独立审查和回滚。

— Qwen Code · qwen3.8-max-preview