failover: retry the same host when no fallback origin exists, and cover the Cloud API hosts - #811
Open
u9g wants to merge 2 commits into
Open
failover: retry the same host when no fallback origin exists, and cover the Cloud API hosts#811u9g wants to merge 2 commits into
u9g wants to merge 2 commits into
Conversation
u9g
requested review from
cloudwebrtc,
lukasIO and
xianshijing-lk
as code owners
September 11, 2026 17:25
u9g
force-pushed
the
jason/cloud-api-retry
branch
2 times, most recently
from
September 11, 2026 17:59
b83ea22 to
5110422
Compare
A retryable failure with no untried origin used to be surfaced after a single attempt. Retry it against the same origin instead, bounded by the existing attempt count and backoff. This matches the cross-region path, which already retries both transport errors and 5xx responses.
u9g
force-pushed
the
jason/cloud-api-retry
branch
from
September 11, 2026 18:00
5110422 to
8448cc3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A request to
cloud-api.livekit.iowas lost between Cloudflare and the origin on 2026-09-11: the edge acknowledged it, no LiveKit system ever saw it, and the client hung until its timeout. The Python SDK's failover could not retry it, for two reasons:is_cloudonly matches*.livekit.cloud, so acloud-api.livekit.iorequest always got exactly one attempt.pick_nextfinds no untried origin, which is always the case for cloud-api: it has a single origin and/settings/regionsreturns 404.What
Two commits, each test-first:
test_region_discovery_unreachableexpectation (region 0 failing with 503 every time raises) is preserved: the 503 surfaces once the attempts are exhausted.is_cloud_api(host)matchescloud-api.<env>.livekit.io(case-insensitively) and joinsis_cloudinfailover_attempts.is_clouditself is unchanged. A Cloud API host has a single origin, so the twirp client never calls region discovery for it; otherwise every failed attempt would also pay the discovery fetch's own timeout.The existing
MIN_FAILOVER_TIMEOUTgate still applies, so requests with a budget under 5 seconds get a single attempt as before.Behavior change
For cloud-api calls with a budget at or above
MIN_FAILOVER_TIMEOUT, a lost request now costs up toFAILOVER_MAX_ATTEMPTS(3) attempts with the existing exponential backoff. A truly dead origin takes up to three times the per-attempt budget to surface instead of one. A single lost request becomes a sub-second blip.Cross-SDK parity
Mirrors livekit/server-sdk-go#1002, including its review follow-ups.
node-sdkshas the same design and the same gap; a separate PR will port it there.Testing
pytest tests/api -qpasses (46 tests) with thelivekit/test-servermock running locally, as in theTest APIworkflow.tests/api/test_failover_unit.pyneeds no mock server:test_retries_same_host_on_transport_error,test_retries_same_host_on_5xx(both against an in-process aiohttp server with no fallback regions),test_cloud_api_host_never_consults_region_discovery(the server is reached ascloud-api.livekit.iothrough a loopback resolver and its/settings/regionshandler records zero hits), and atest_failover_attemptstable with the cloud-api rows. Each was confirmed failing before its implementation commit.ruff checkandruff format --checkpass with the repo's pinned ruff (0.15.4).