stdio MCP servers accumulate under a live app-server on 26.810.52044, after the #18881 / #19753 shutdown fix

[twkjtwkj]

Filing separately from the closed #18881 because the shutdown path that PR #19753
fixed now works on this build, while accumulation during a live session does
not. Cross-referencing #12491 (open, GUI), #20349, #25015, #26984.

Environment

Codex (ChatGPT.app)	26.810.52044, bundle 6662
codex-cli	0.148.0-alpha.9
macOS	26.6.1 (25G76), arm64
Surface	Codex desktop app, codex ... app-server

Two stdio MCP servers configured in ~/.codex/config.toml, both launched via a
wrapper script that execs (so the wrapper leaves no process of its own):

[mcp_servers.fin]
command = "/Users/<user>/.ai_ops/bin/mcp-launch.sh"
args = ["fin"]          # execs a python stdio server

[mcp_servers.email]
command = "/Users/<user>/.ai_ops/bin/mcp-launch.sh"
args = ["email"]        # execs `npx -y @codefuturist/email-mcp stdio`

What happens

A long-lived app-server spawns a fresh stdio MCP server roughly every three
minutes and never closes the previous one. The old servers stay parented to the
app-server with their stdio pipes still held open (lsof shows fds 0/1/2 as
PIPE), at 0.0% CPU and ~1s of accumulated CPU time. They are idle but retained.

This is the same shape as #18881, but that issue was closed by PR #19753
(merged 2026-04-28) and this build postdates it.

What PR #19753 did fix, verified here

Shutdown draining works. When the app-server exits, its MCP children go with it.
Observed directly: app-server PID 43065 was restarted, and all ~33 of its
accumulated MCP children terminated along with it, requiring no manual cleanup.

So this is not a regression of the shutdown path. It is the in-session path,
where servers are replaced but the superseded ones are never shut down.

Measurements

Two independent windows, one app-server each, machine otherwise idle:

Rate	~1 new stdio server per 3 minutes, sustained
Single app-server (PID 14353), 71 min uptime	24 fin_mcp children
Across both configured servers, ~1 hour after a manual clear from 11 processes	137 processes
RSS at that point	~5.4 GB
Free system memory at that point	141 MB
After killing the superseded children	4480 MB free

Reproduced twice, hours apart, across an app-server restart in between.

Reproduction

1. Configure one or more stdio MCP servers in ~/.codex/config.toml.
2. Start the Codex desktop app and leave a project session open.
3. Watch the children of the app-server process:

APPSRV=$(pgrep -f "Resources/codex .*app-server" | head -1)
watch -n 30 "pgrep -P $APPSRV -f mcp | wc -l"

The count climbs monotonically and never decreases while the app-server lives.

Expected

When a stdio MCP server is replaced, the superseded one is shut down and its
process reaped, so the count tracks the number of configured servers rather than
session age.

Notes

Each server is cheap alone (~28 to 90 MB here) and the fault is only visible over
hours. With two servers configured it took roughly one hour to consume several GB
and drive the machine to 141 MB free. #12491 reports the same end state at much
larger scale (1319 processes, 37 GB), which suggests the ceiling is however long
the app stays open.

The three fix directions proposed in #12491 (process groups, startup reaping of
stale trees, heartbeat self-termination) would each cover this case too. The
narrowest fix specific to what is seen here is shutting down the superseded
server at the point of replacement, rather than only at session shutdown, which
is what #19753 addressed.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• [Windows Codex app] Local stdio MCP servers are repeatedly spawned and not reaped within a single task #38754
• Windows Codex App repeatedly creates and retains hundreds of duplicate Node/MCP processes during a single task #38825

Powered by Codex Action

[jdcodes1]

Traced the in-session path on main @ 1f41cc5, and the code structure matches your observation precisely: the shutdown that #19753 added exists only on the session-end path; the in-session replacement path has no shutdown at all, and its Drop-based fallback is a no-op for exactly the processes you're seeing accumulate.

1. Replacement never shuts superseded servers down. A refresh goes through McpRuntime::replace → publish, which builds a new McpConnectionSet (reusing previous connections only when their identity matches) and then just swaps the published pointer:

codex/codex-rs/codex-mcp/src/runtime.rs

Lines 183 to 231 in 1f41cc5

	/// Reconciles configured servers and publishes their immutable runtime snapshot.
	pub async fn replace(&self, input: McpRuntimeInput) {
	let current = self.current.load_full();
	let mut reconnect = McpReconnectGuard {
	pending: &self.reconnect_pending,
	claimed: self.reconnect_pending.swap(false, Ordering::AcqRel),
	};
	self.publish(
	input,
	(!reconnect.claimed).then_some(current.connections.as_ref()),
	)
	.await;
	reconnect.claimed = false;
	}
	/// Starts fresh connections and returns their complete, refreshed Apps catalog.
	pub async fn replace_fresh(&self, input: McpRuntimeInput) -> anyhow::Result<Vec<ToolInfo>> {
	self.publish(input, /*previous*/ None).await;
	self.latest_hard_refresh_codex_apps_tools_cache().await
	}
	async fn publish(&self, input: McpRuntimeInput, previous: Option<&McpConnectionSet>) {
	let (publish, publication_gate) = McpPublicationGate::pending();
	let config = Arc::clone(&input.config);
	let auth = input.auth.clone();
	let auth_token = auth.as_ref().and_then(|auth| auth.get_token().ok());
	let plugins_available = input.plugins_available;
	let ready_selected_capability_roots = input.ready_selected_capability_roots.clone();
	let connections = Arc::new(
	McpConnectionSet::new(
	previous,
	publication_gate,
	input,
	self.elicitation_router.clone(),
	)
	.await,
	);
	self.current.store(Arc::new(PublishedMcpRuntime {
	connections,
	config: Some(config),
	auth,
	auth_token,
	plugins_available,
	ready_selected_capability_roots,
	cached_binding: Mutex::new(None),
	}));
	let _ = publish.send(true);
	}

The proper cleanup — McpConnectionSet::shutdown, which terminates each stdio child (connection_manager.rs#L813-L828) — is invoked from exactly one place: McpRuntime::shutdown (runtime.rs#L441), i.e. the path #19753 fixed. Superseded connections on the replace path are simply dropped.

2. The Drop fallback cannot kill a started server. Drop for McpServerConnection only cancels the startup token:

codex/codex-rs/codex-mcp/src/connection_manager.rs

Lines 148 to 152 in 1f41cc5

	impl Drop for McpServerConnection {
	fn drop(&mut self) {
	self.client.cancel_token.cancel();
	}
	}

and that token only guards the startup future (.or_cancel(...) in rmcp_client.rs#L370). Once startup_complete is set, cancelling it does nothing. Actual child termination happens in StdioServerProcessHandleInner::drop / terminate() (stdio_server_launcher.rs#L497+), which requires the last Arc<RmcpClient> clone to go away — and clones live inside the Shared startup future, cached bindings, and tool-catalog plumbing. Your lsof evidence (superseded children with all three stdio pipes still open in the live app-server) demonstrates that some clone does survive in practice, so the children are never terminated until process exit. This is why the leak is monotonic while the app-server lives yet fully drains on restart.

3. Why a fresh server spawns at all (the ~3-minute cadence). Reuse requires the new McpServerConnectionIdentity to equal the old one, and the identity includes volatile inputs — notably the raw auth token (runtime_auth_token: Option<String>) and the resolved values of referenced environment variables:

codex/codex-rs/codex-mcp/src/server.rs

Lines 95 to 110 in 1f41cc5

	pub(crate) struct McpServerConnectionIdentity {
	transport: McpServerTransportConfig,
	environment_id: String,
	oauth_store: Option<(OAuthCredentialsStoreMode, AuthKeyringBackendKind)>,
	oauth_credentials: Result<Option<StoredOAuthCredentialSnapshot>, String>,
	pub(crate) oauth_store_was_contended: bool,
	resolved_environment: Result<Option<Arc<Environment>>, String>,
	local_stdio_fallback_cwd: Option<PathBuf>,
	referenced_environment_variables: Vec<(String, Option<OsString>)>,
	runtime_auth: Option<CodexAuth>,
	runtime_auth_token: Option<String>,
	codex_apps_cache_identity: Option<(PathBuf, ConnectorRuntimeContextKey)>,
	client_elicitation_capability: ElicitationCapability,
	client_mcp_extensions: ClientMcpExtensions,
	agent_plugin: bool,
	}

Meanwhile the session's MCP prewarm worker marks the runtime dirty and re-publishes on every auth-change tick of the auth manager's watch channel (start_mcp_prewarm_worker in core/src/session/mcp_prewarm.rs). So: periodic token refresh → watch tick → refresh → identity mismatch (token string changed) → brand-new child, old one orphaned by (1)+(2). A testable prediction for your setup: stat -f %m ~/.codex/auth.json should advance at the same ~3-minute cadence as new fin_mcp children appear. If it does, that's the trigger confirmed end-to-end.

Fix outline.

1. In publish, diff the previous set against the new one and explicitly shutdown() every previous connection that wasn't carried over — the in-session analog of Terminate stdio MCP servers on shutdown to avoid process leaks #19753, ideally spawned as a detached task like set-level shutdown already does so an interrupt can't cancel cleanup.
2. Harden Drop for McpServerConnection to schedule a full shutdown() for startup-complete clients instead of only cancelling the startup token, so no future refactor can silently reintroduce the leak.
3. Reduce the churn itself: for stdio servers the spawned child's command/env doesn't change when the ambient auth token rotates, so the token belongs in the identity only for transports that actually consume it (HTTP bearer, Codex Apps). Comparing spawn-relevant inputs only would make token rotation reuse the existing child.
4. Log respawns with the identity field that differed — this issue took real forensics to see; a single info! line would have surfaced it immediately.

[twkjtwkj]

Windows 11 repro, Codex desktop 26.810.7004.0 (ChatGPT.exe hosting codex.exe), 8 stdio servers, ChatGPT auth.

Fresh boot, one Codex thread, no authentication rotation or config changes. Codex started at 11:12:04. By 11:25:46, the same codex.exe had 70 MCP descendant processes using 1,924 MB. It retained three chains each for Azure DevOps, ServiceNow, and Power BI, plus six Snowflake chains for two configured Snowflake servers. This indicates startup created and retained three complete MCP server sets for a single thread.

Earlier the same day, over a 35-minute watched run of MCP tool calls: auth.json mtime never changed (Aug 10 17:42 throughout), a fourth complete set spawned at the next turn boundary after a config.toml edit was reverted, and one superseded set stayed alive beside the replacement for the rest of the run. So on this platform the leak reproduces without any auth change; each publish leaves the previous set alive, matching point (1). Point (3) alone would not prevent it.

[ryofukutani]

macOS最新版でも同じlive app-server内のMCPランタイム蓄積を再現しました。

環境:

• ChatGPT.app 26.814.41407
• bundled codex-cli 0.148.0-alpha.15
• macOS 26.5.2 arm64
• 32GB Apple Silicon Mac

観測結果（2026-08-19 JST）:

• ChatGPT.app本体は約360MBだが、子のcodex ... app-server --analytics-default-enabledはRSS最大約5.1GB、physical footprint peak 4.7GB。
• app-server配下の直接子プロセスは最大53個で、node_repl、gitnexus、Slack、Context7、Pencil、mem0などの同じstdio MCP群が複数世代存在した。
• 各MCPプロセスは個別にはほぼidle（CPU 0%前後）だが、同じapp-serverの下に時間差で4グループ以上残っていた。
• app-serverは/Volumes/RAID/cache/codex_sessionsのrollout JSONLを複数開いており、2026-07-23開始のファイルが約2.03GiBまで増加し、調査中も追記されていた。
• app-serverのCPUは100〜300%に達し、システムswapは92〜98%使用、残量は約0.5〜1.3GiBまで低下した。
• app-serverを再起動すると、蓄積したMCPプロセス群はまとめて消えるため、MCP個別実装の常時CPUリークというより、live app-serverのthread/MCP runtime保持と履歴再水和の問題と考えている。

補足:

• ChatGPT.app内蔵のapp-serverバイナリはMach-O arm64で、rustc/tokioの痕跡があり、Rust化は根本対策にならない。
• 期待動作は、threadがinactive/archive/unsubscribeになった時点でMCP runtimeだけをdetach/shutdownし、履歴は保持してもstdioプロセスと巨大履歴の再水和を無制限に保持しないこと。
• 既存の#38925にある「superseded connectionの明示shutdown」「spawn-relevant identityだけで再利用」「idle runtimeの上限/TTL」という修正案と一致する。

再現時のcredential・会話本文は共有していません。

(codex-lead レーンによる報告です)

[ryofukutani]

追加のアプリ層比較です（同一macOSホスト上の読み取り専用プロセス集計）。

• ChatGPT.app本体の子孫: 112プロセス、合計RSS約9.1GB
• そのapp-server単体の子孫: 102プロセス、合計RSS約7.7GB
• Claude.app本体の子孫: 12プロセス、合計RSS約310MB
• Claude側のCLI/MCPは別プロセスツリーとして分離され、ChatGPTのapp-serverのように全スレッドのMCPを一つの長寿命親へ集約していない

この比較から、個別MCPの実装だけではChatGPT.appだけで肥大する差を説明できません。ChatGPT.appのアプリ層が、複数threadのruntimeとMCP stdioプロセスを単一の長寿命app-serverへ保持する設計・ライフサイクルが増幅要因です。

期待する修正境界は、GUIのrenderer最適化ではなく、app-serverのthread unload / unsubscribe / MCP runtime ownershipです。論理threadのresume用状態を保持しても、MCP runtimeとstdio子プロセスは切り離して、必要時にlazy再生成できるべきです。

(codex-lead レーンによる追加監査です)