fix(core): prevent worker metrics loss when cache_tracker init fails

[m199369309]

Summary

• Worker-level metrics (workers_total, _worker_status-based gauges) were persistently empty because get_supervisor_ref coupled add_worker registration with record_model_version cache sync inside a single try block. When record_model_version raised, _clear_supervisor_refs wiped self._supervisor_ref, so report_status / heartbeat could never send worker status — even though the supervisor saw workers_total=4.
• Decouple the two concerns: registration is tracked by an explicit _registered flag, cache sync is isolated in its own try block and treated as an auxiliary feature whose failure must not tear down the worker↔supervisor channel.
• Harden cache_tracker so a version-count mismatch between supervisor and worker becomes a logged skip rather than an AssertionError that aborts the whole sync path.
• Isolate record_model_version in supervisor.register_model / worker.register_model so an auxiliary-cache failure does not roll back the already-successful register_fn. register_fn failures do roll back via unregister_fn (preserving the origin/main safety guarantee), and _sync_register_model failures do roll back + propagate (preserving cluster-state consistency).

Changes

• xinference/core/worker.py
  • New self._registered: bool flag tracking whether supervisor.add_worker() has succeeded.
  • get_supervisor_ref cache-hit path returns early only when no registration is required, or the worker is already registered; otherwise falls through to run add_worker.
  • record_model_version moved into its own try block with a defensive _cache_tracker_ref is None guard; failures are logged as warnings, not propagated.
  • _clear_supervisor_refs resets _registered to keep it in sync with the _supervisor_ref lifecycle.
  • register_model: register_fn failure rolls back via unregister_fn (restored per maintainer feedback); record_model_version failure is warning-only.
• xinference/core/cache_tracker.py
  • record_model_version: replace AssertionError on version-count mismatch with a logger.warning + continue, so cache sync degrades instead of crashing.
• xinference/core/supervisor.py
  • register_model: split the single try block into (1) local register_fn (rolls back on failure), (2) record_model_version cache sync (warning-only, no rollback), (3) _sync_register_model worker propagation (rolls back register_fn and re-raises on failure).
• xinference/core/tests/test_worker.py
  • set_supervisor_ref_for_test now also sets _registered = True so tests that inject a fake supervisor ref (without going through add_worker) still hit the cache-hit path in get_supervisor_ref(add_worker=True).

Root cause

get_supervisor_ref previously did:

try:
    supervisor_ref = await xo.actor_ref(...)
    if add_worker:
        await supervisor_ref.add_worker(...)
    # ... build model_version_infos ...
    await self._cache_tracker_ref.record_model_version(...)  # ← failure here
except Exception:
    self._clear_supervisor_refs()  # ← wiped _supervisor_ref
    raise

So any failure in record_model_version (e.g. transient AssertionError from a version-count mismatch on a re-reporting worker) cleared _supervisor_ref, blocking heartbeat and report_status for the worker. workers_total would still count the worker (because add_worker had already run), but every gauge sourced from _worker_status would stay empty.

Test plan

• black --check / flake8 / isort on modified files — clean.
• pytest xinference/core/tests/test_worker.py -k test_report_status — verifies the injected-ref cache-hit path works with _registered=True. (Cannot run locally due to missing torchvision; CI will validate.)
• Unit test: get_supervisor_ref with add_worker=True calls supervisor.add_worker exactly once across retries when record_model_version raises.
• Unit test: _cache_tracker_ref is None does not raise AttributeError from get_supervisor_ref.
• Unit test: cache_tracker.record_model_version with mismatched version counts logs a warning and continues (no AssertionError).
• Unit test: supervisor.register_model / worker.register_model rolls back local registration (calls unregister_fn) and re-raises when register_fn fails.
• Unit test: supervisor.register_model rolls back and re-raises when _sync_register_model fails.
• Manual: on a multi-worker cluster, kill+restart cache_tracker; verify worker status gauges stay populated and supervisor logs a warning rather than dropping the worker.

Dependencies

None. This PR is independent of the dashboard-split work tracked separately.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces changes to improve robustness and error handling during model registration and cache synchronization across the supervisor and workers. Key changes include adding defensive checks for _cache_tracker_ref being None, introducing a _registered flag to track worker registration state, and logging warnings instead of failing when auxiliary cache operations fail. However, two critical issues were identified in the review: first, suppressing failures in _sync_register_model on the supervisor can lead to inconsistent cluster states and silent launch failures, so registration should be rolled back and the error propagated; second, removing asyncio.to_thread from the requests.get call in worker.py blocks the main event loop, which could trigger heartbeat timeouts and falsely mark workers as dead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qinxuye]

Two issues need cleanup before this can land.

[qinxuye]

LGTM