Skip to content

feat: TCP keepalive, reverse-channel probe, list_models cache, .pth atomic write, launch exemption - #4845

Merged
qinxuye merged 3 commits into
xorbitsai:mainfrom
m199369309:fix/keepalive-reverse-ping-pth-atomic-launch-exempt
Apr 26, 2026
Merged

feat: TCP keepalive, reverse-channel probe, list_models cache, .pth atomic write, launch exemption#4845
qinxuye merged 3 commits into
xorbitsai:mainfrom
m199369309:fix/keepalive-reverse-ping-pth-atomic-launch-exempt

Conversation

@m199369309

@m199369309 m199369309 commented Apr 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Require xoscar>=0.9.6 for native TCP keepalive support (BUG: enable TCP keepalive for SocketClient to detect dead connections xoscar#189), removing the xinference-side monkey-patch
  • Add supervisor→worker reverse-channel ping probe in _check_dead_nodes to detect dead reverse channels even when worker→supervisor heartbeat is healthy
  • Add list_models cache degradation: return cached results when a worker is temporarily unreachable instead of returning empty dict (prevents UI from showing no models)
  • Reduce XINFERENCE_GET_MODEL_RPC_TIMEOUT from 120s to 30s to limit actor lock contention during worker communication failures
  • Fix .pth file concurrent write race in virtualenv setup by using atomic write (os.replace) and skipping writes when content is unchanged
  • Exempt workers from reverse-channel dead detection while launching models, as long-running model downloads can starve the actor event loop causing false-positive dead detection
  • Limit huggingface_hub download concurrency to 2 (XINFERENCE_MODEL_DOWNLOAD_WORKERS) to reduce GIL contention during model downloads

Changed Files

File Change
setup.cfg xoscar>=0.9.3 → xoscar>=0.9.6 (native TCP keepalive)
xinference/core/supervisor.py Reverse-channel probe, list_models cache, _workers_launching tracking
xinference/core/worker.py ping() method, .pth atomic write, download concurrency limit
xinference/constants.py RPC timeout 120→30, new XINFERENCE_MODEL_DOWNLOAD_WORKERS

Test plan

  • CI pipeline passes
  • Verify TCP keepalive active via xoscar native: ss -tnop | grep <supervisor-port> shows timer:(keepalive,...)
  • Verify reverse-channel detection: simulate network partition triggers "Worker reverse-channel timeout" within 25s
  • Verify list_models degradation: worker unreachable → UI still shows cached models
  • Verify launching exemption: deploying large model does not trigger false "Worker dead"
  • Verify .pth race fix: multi-replica deploy on existing venv succeeds on first attempt

…tomic write, launch exemption

- Add xoscar TCP keepalive monkey-patch (SO_KEEPALIVE + KEEPIDLE=60s)
  to prevent idle connections from being silently dropped by firewalls
- Add supervisor->worker reverse-channel ping probe in _check_dead_nodes
  to detect dead reverse channels even when heartbeat is healthy
- Add list_models cache degradation: return cached results when a worker
  is temporarily unreachable instead of returning empty dict
- Reduce XINFERENCE_GET_MODEL_RPC_TIMEOUT from 120s to 30s to limit
  actor lock contention during worker communication failures
- Fix .pth file concurrent write race in virtualenv setup by using
  atomic write (os.replace) and skipping writes when content unchanged
- Exempt workers from reverse-channel dead detection while launching
  models, as long downloads can starve the actor event loop
- Limit huggingface_hub download concurrency to 2 (XINFERENCE_MODEL_DOWNLOAD_WORKERS)
  to reduce GIL contention during model downloads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@XprobeBot XprobeBot added this to the v2.x milestone Apr 24, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread xinference/core/xoscar_keepalive.py Outdated
xoscar v0.9.6 natively implements TCP keepalive in SocketClient.connect()
with identical parameters (KEEPIDLE=60s, KEEPINTVL=10s, KEEPCNT=3) and
additionally covers Windows. The xinference monkey-patch is now redundant.

- Upgrade xoscar minimum version from 0.9.3 to 0.9.6
- Delete xinference/core/xoscar_keepalive.py
- Remove keepalive import from deploy/{worker,supervisor,local}.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@qinxuye qinxuye left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qinxuye
qinxuye merged commit 1cd4f79 into xorbitsai:main Apr 26, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants