fix: refresh runtime-proxy credentials before pruning a session on 404/401 - #123
Open
zoutei wants to merge 1 commit into
Open
fix: refresh runtime-proxy credentials before pruning a session on 404/401#123zoutei wants to merge 1 commit into
zoutei wants to merge 1 commit into
Conversation
…4/401 The runtime proxy token is short-lived and expires well before an assignment is torn down, so an expired token produces the exact same 404/401 that a genuinely terminated runtime would. `exec`/`repl`/ `console`/`run` treated that error as proof the session was lost and deleted the local binding outright -- even though `list_assignments()` keeps listing the assignment (and keeps minting it a fresh proxy token) long after the stored token has stopped working. A live, billing runtime was being destroyed by its own client roughly once per token lifetime on any long-running session. `State.prune_or_recover_session()` asks the server before deleting: if the endpoint is still listed, adopt the fresh token/url and keep the session (this call still reports failure, but the next one succeeds); if the endpoint is confirmed gone, prune as before; if the control plane can't be reached at all, keep the session rather than delete on an inconclusive check. Fixes googlecolab#106.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The runtime-proxy token is short-lived, roughly 60 minutes (the server reports a 3600-second lifetime via
tokenExpiresInSeconds), and expires well before an assignment is actually torn down. An expired token produces the exact same 404/401 that a genuinely terminated runtime would.exec,repl,console, andruntreated any 404/401 as proof the session was lost and deleted the local binding outright, even thoughlist_assignments()keeps listing the assignment and keeps minting it a fresh proxy token long after the previously-stored token stops working. A live, still-billing runtime could be destroyed by its own client roughly once per hour on any session kept open longer than that.Fixes #106.
What changed
State.prune_or_recover_session()replaces the unconditional prune on this error path. Before deleting anything it asks the server. If the endpoint is still listed, it adopts the fresh token and url and keeps the local session; this invocation still fails, but the next one succeeds with the refreshed credential. If the endpoint is confirmed gone, it prunes as before. If the control plane is unreachable, it preserves the session rather than delete on an inconclusive check. This only runs reactively, in response to an actual 404/401 from the runtime proxy; there's no background or proactive token refresh.Testing
Full test suite passes. Added unit coverage for all three outcomes of
prune_or_recover_session, plus CLI-level tests covering both the pruned and preserved paths forexec.