Skip to content

failover: retry the same host when no fallback origin exists, and cover the Cloud API hosts - #811

Open
u9g wants to merge 2 commits into
mainfrom
jason/cloud-api-retry
Open

failover: retry the same host when no fallback origin exists, and cover the Cloud API hosts#811
u9g wants to merge 2 commits into
mainfrom
jason/cloud-api-retry

Conversation

@u9g

@u9g u9g commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Why

A request to cloud-api.livekit.io was lost between Cloudflare and the origin on 2026-09-11: the edge acknowledged it, no LiveKit system ever saw it, and the client hung until its timeout. The Python SDK's failover could not retry it, for two reasons:

  • is_cloud only matches *.livekit.cloud, so a cloud-api.livekit.io request always got exactly one attempt.
  • Even with failover engaged, the retry loop raises as soon as pick_next finds no untried origin, which is always the case for cloud-api: it has a single origin and /settings/regions returns 404.

What

Two commits, each test-first:

  1. Retry the same host when a retryable failure has no fallback origin. When no untried origin exists, a retryable failure (a transport error or a 5xx) is re-sent to the same origin instead of being raised, until the attempts are exhausted. A transport error can also mean the server executed the request and the response was lost, so treating 5xx differently would buy no idempotency safety, and cross-region failover already retries 5xx. The existing test_region_discovery_unreachable expectation (region 0 failing with 503 every time raises) is preserved: the 503 surfaces once the attempts are exhausted.
  2. Enable retries for the LiveKit Cloud API hosts. A new is_cloud_api(host) matches cloud-api.<env>.livekit.io (case-insensitively) and joins is_cloud in failover_attempts. is_cloud itself is unchanged. A Cloud API host has a single origin, so the twirp client never calls region discovery for it; otherwise every failed attempt would also pay the discovery fetch's own timeout.

The existing MIN_FAILOVER_TIMEOUT gate still applies, so requests with a budget under 5 seconds get a single attempt as before.

Behavior change

For cloud-api calls with a budget at or above MIN_FAILOVER_TIMEOUT, a lost request now costs up to FAILOVER_MAX_ATTEMPTS (3) attempts with the existing exponential backoff. A truly dead origin takes up to three times the per-attempt budget to surface instead of one. A single lost request becomes a sub-second blip.

Cross-SDK parity

Mirrors livekit/server-sdk-go#1002, including its review follow-ups. node-sdks has the same design and the same gap; a separate PR will port it there.

Testing

  • pytest tests/api -q passes (46 tests) with the livekit/test-server mock running locally, as in the Test API workflow.
  • New tests/api/test_failover_unit.py needs no mock server: test_retries_same_host_on_transport_error, test_retries_same_host_on_5xx (both against an in-process aiohttp server with no fallback regions), test_cloud_api_host_never_consults_region_discovery (the server is reached as cloud-api.livekit.io through a loopback resolver and its /settings/regions handler records zero hits), and a test_failover_attempts table with the cloud-api rows. Each was confirmed failing before its implementation commit.
  • ruff check and ruff format --check pass with the repo's pinned ruff (0.15.4).

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@u9g
u9g force-pushed the jason/cloud-api-retry branch 2 times, most recently from b83ea22 to 5110422 Compare September 11, 2026 17:59
u9g added 2 commits September 11, 2026 14:00
A retryable failure with no untried origin used to be surfaced after a
single attempt. Retry it against the same origin instead, bounded by the
existing attempt count and backoff. This matches the cross-region path,
which already retries both transport errors and 5xx responses.
@u9g
u9g force-pushed the jason/cloud-api-retry branch from 5110422 to 8448cc3 Compare September 11, 2026 18:00
@u9g u9g changed the title failover: retry a lost request on the same host, and cover the Cloud API hosts failover: retry the same host when no fallback origin exists, and cover the Cloud API hosts Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant