fix(runtime): reclaim orca-runtime.json when it stops describing this runtime - #10840
Conversation
… runtime On macOS the Chromium single-instance lock is silently defeated whenever `SingletonSocket`/`SingletonCookie` go missing from the profile — and the socket they point at lives under `$TMPDIR` (`/var/folders/.../T`), which macOS purges after 3 days (`com.apple.bsd.dirhelper`, CLEAN_FILES_OLDER_THAN_DAYS=3). A launch that slips past the lock runs a full startup, republishes `orca-runtime.json` with its own pid, and leaves the CLI on a dead pid once it exits: `orca status` reports `stale_bootstrap` and every terminal command fails `runtime_unavailable` while the original app keeps serving. The owner now watches its own discovery record and republishes once no live runtime is described. Reclaiming only a dead pid is deliberate: two live runtimes sharing a profile would otherwise fight over the file. Reproduced on macOS with two real Orca main processes on one profile: the second instance took the lock and clobbered the record, and killing it left `stale_bootstrap` against the still-healthy first instance. With this change the owner reclaimed the record in ~2s and the CLI returned to `ready`. Refs #7848
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds runtime metadata ownership detection and recovery. The watcher evaluates metadata identity and PID liveness, polls for changes, handles unreadable records, retries failed republishing, and exposes check/stop controls. The RPC server starts the watcher after publishing metadata and stops it during shutdown. Tests cover reclaim decisions, polling behavior, republishing failures, live sibling runtimes, and stopped servers. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 38f30a93-d727-40fe-8dcf-244cbca2e979
📒 Files selected for processing (4)
src/main/runtime/runtime-metadata-ownership-watch.test.tssrc/main/runtime/runtime-metadata-ownership-watch.tssrc/main/runtime/runtime-rpc.test.tssrc/main/runtime/runtime-rpc.ts
The republish guard alone kept the shutdown test green, so the watch teardown was unasserted. Also drop the doc claim of startup/activation callers that do not exist.
Windows never assigns pid 1, so the hardcoded sibling read as dead there and the watch would reclaim the record. Own a synthetic pid instead and let process.pid play the live sibling.
Addresses #7848 (STA-1513) — the macOS manifestation. Companion to #10463, which fixes the Linux one. Different mechanism, no file overlap; both are needed.
Problem
On macOS a second launch does not lose the single-instance lock — it can win it. Chromium's lock depends on the profile's
SingletonSocket/SingletonCookie, and the socket those point at lives under$TMPDIR(/var/folders/.../T). When those entries go missing, a second launch acquires the lock while the owner is alive, completes startup, and republishesorca-runtime.jsonwith its own pid. Once it exits, the CLI is pointed at a dead pid while the original app keeps serving —stale_bootstrap/app.running: false/runtime_unavailableagainst a healthy app.This is not the lock-loser path #10463 guards: that PR's guard does not apply here, because the second instance genuinely holds the lock.
Reproduction
Probe matrix, owner alive throughout (S1 is the control):
lock=false, quits pre-readyrm -rfthe$TMPDIR/scoped_dir*holdingSingletonSocketlock=true, runswhenReadySingletonLock→OtherHost.local-<pid>lock=false, quits pre-readySingletonSocketfile, keep its dirlock=trueSingletonSocketsymlinklock=trueSingletonCookiesymlinklock=trueConfirmed with two real Orca main processes on one isolated profile, not just the probe:
Byte-for-byte the reported output, including at
v1.4.121(the reported version).Why the files go missing (hypothesis, not observed):
com.apple.bsd.dirhelperruns daily at 03:35 withCLEAN_FILES_OLDER_THAN_DAYS=3over the per-user/var/folders/*/*/Ttree, by access time. A live Orca'sSingletonSocketis only accessed when a second launch tries to notify it, so on a Mac left running 3+ days it is a purge candidate. Orca never touchesSingleton*itself (grepped) — the deletion always comes from outside the app. I simulated the deletion; I did not watch macOS perform it.Fix
Rather than trying to enumerate every way the lock can be defeated, the runtime owner now watches its own discovery record and reclaims it when it stops describing a live runtime. This is mechanism-agnostic: it repairs the record regardless of what clobbered it.
Ping-pong safety is the critical property: reclaim happens only when the recorded pid is not running, so two live runtimes sharing a profile leave each other alone.
isPidRunningtreats onlyESRCHas dead (EPERMmeans a foreign owner holds the pid), matching the existing socket-sweep rule. Live-verified: 25s of coexistence with no ping-pong, then reclaim ~2s afterkill -9.Shutdown is covered twice:
stop()clearsactiveTransportsbefore stopping the watch, andrepublishno-ops when there are no active transports, so a poll landing mid-teardown cannot resurrect a record. The timer isunref'd so discovery bookkeeping never keeps the process alive.Ruled out
Hostname change in
SingletonLock(S3), websocket-only / null-authToken(thestatus.tsPath 1 branch), the 104-bytesun_pathlimit, and corrupt records. A genuine lock loser on macOS still dies pre-ready (S1), which re-confirms the platform fact established in #10463.Known gap, deliberately not fixed here
src/main/index.ts:2456still swallows aruntimeRpc.start()rejection on the desktop path. The new watch does not cover it, because the watch arms only after a successful publish. An app that runs with no discovery record at all is a separate robustness bug and deserves its own change.Test plan
runtime-metadata-ownership-watchunit tests: leaves its own record alone, reclaims missing/dead/unreadable records, reclaims a foreign runtimeId stamped on this pid, yields to another live runtime so two instances cannot ping-pong, polls without an explicit check, stops onstop(), and keeps polling after a republish failure.runtime-rpctests for republish-after-clobber and delete-underneath.vitest src/main/runtime/→ 2667 passed, 0 failed. typecheck + oxlint clean.Full investigation report:
docs/bug-reproductions/sta-1513-macos-stale-bootstrap.md.