Fix of Model HTTP retry/backoff + real background-job cancellation#129
Open
anon13377331 wants to merge 3 commits into
Open
Fix of Model HTTP retry/backoff + real background-job cancellation#129anon13377331 wants to merge 3 commits into
anon13377331 wants to merge 3 commits into
Conversation
OpenAiResponsesAgentAdapter now routes every Responses call through one HTTP path that retries 429/5xx with bounded exponential backoff plus jitter, honors Retry-After (seconds or HTTP-date), and polls for cancellation while backing off. runTask also catches RuntimeException so a misbehaving upstream surfaces as a failed task result instead of an uncaught exception. OpenPhoneAgentJobScheduler wires ToolExecutor.isCancelled() to the job store: a run counts as cancelled once its stored status leaves "running" (background_job_stop or stuck-run repair), and a watchdog thread hard-cancels the adapter so a stopped job stops even while blocked on a stuck upstream read. Job runner threads catch RuntimeException and record a failed run with backoff instead of crashing the assistant process. AgentJobStore gains statusOf() and live-run guards on markCompleted and markFailed so a finishing or failing run cannot resurrect a job the user already stopped; notifications follow the recorded outcome. Validation: scripts/check.sh passes (including check-assistant-java against android-36); a local JVM harness drove the real adapter against a stub HTTP server covering 429-then-success, Retry-After preference, bounded attempts on persistent 500, no retry on 400, and cancel during backoff. No device eval was run (no device available in this environment). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASVb6mqLiGd7naEnLaoKnX
Serialize all AgentJobStore mutations on a process-wide lock: every caller constructs its own store over the same SharedPreferences file, so the previous instance-level synchronized let a concurrent background_job_stop be erased by a job thread committing a stale jobs array (read-modify-write race). markCompleted/markFailed/markDispatched now also require a run-ownership token (the markRunning timestamp) so a zombie run reclaimed by stuck-run repair cannot record over its successor, and a stop that lands during runtime dispatch wins. The cancel watchdog now keeps re-cancelling every poll until the run finishes instead of exiting after the first cancel(), and the adapter checks cancellation before opening a request and again after publishing the connection, closing the window where a cancel saw no connection to disconnect and the following stuck read had nothing to break it. Background runs that end in an infrastructure error (network_error from exhausted retries, adapter_error from an unchecked crash) are now recorded as failures with the job store's backoff and repeated-failure alerting instead of counting as completed runs. A Retry-After longer than the in-call retry budget fails the call immediately rather than retrying inside the server's declared closed window. Backoff deadlines and cancellation-poll throttling use the monotonic clock, and scheduleNext keeps the alarm armed while any job is marked running so a run stranded by process death is repaired without waiting for a boot. Validation: scripts/check.sh passes (javac gate against android-36); the local JVM harness now also covers fail-fast on Retry-After beyond the retry budget, alongside 429-then-success, Retry-After preference, bounded attempts on persistent 500, no retry on 400, and cancel during backoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASVb6mqLiGd7naEnLaoKnX
…ation-91oqc5 Model HTTP retry/backoff and real background-job cancellation
|
@anon13377331 is attempting to deploy a commit to the adam-7512's projects Team on Vercel. A member of the Team first needs to authorize it. |
Author
|
please tell me if there is anything else I can help with on my side (currently waiting for approval of the workflow run-gcp-lab) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #88 (Model HTTP retry/backoff + real background-job cancellation).
OpenAiResponsesAgentAdapternow funnels all Responses API calls through a singlepostResponsespath (removing three duplicated HTTP blocks) that retries 429/5xx up to 3 attempts with exponential backoff plus jitter, honorsRetry-After(both delta-seconds and HTTP-date forms), and fails fast whenRetry-Afterexceeds the 30s in-call budget. Backoff waits poll for cancellation on a monotonic clock, and non-transient statuses (e.g. 400) are not retried. Timeouts and error-message format are unchanged.OpenPhoneAgentJobScheduler.isCancelled()no longer hardcodesfalse: each run gets aJobCancellationSignalbacked by the job store, keyed by a run-ownership token (themarkRunningtimestamp).background_job_stop(or stuck-run repair) makes the signal fire; a watchdog thread then repeatedly callsadapter.cancel()so even a thread blocked on a stuck upstream read is torn down. AllAgentJobStoreread-modify-write cycles are serialized on a process-wide lock (callers construct independent instances over one SharedPreferences file, so instance-levelsynchronizedprovided no exclusion), andmarkCompleted/markFailed/markDispatchedrefuse to record for a run that is no longer live — a stopped job can't be resurrected or notify as finished.network_error,adapter_error) now follow failure backoff and repeated-failure alerting rather than counting as completions.scheduleNextkeeps the alarm armed while any job is marked running so a run stranded by process death is repaired without waiting for a boot.docs/AGENT_RUNTIME_V1.mdgains a "Cancellation and Upstream Failures" section; changelog updated under Unreleased.Validation
./scripts/check.sh— passes, including the assistantjavacgate (86 files against android-36) and the broker smoke testgit diff --check— cleanrun-gcp-labemulator run for on-device validation (this change affects background-job behavior).Retry-Afterpreference, fail-fast onRetry-Afterbeyond the budget, bounded attempts on persistent 500, no retry on 400, and cancel-during-backoff stopping within ~1s and producing the "stopped" reply. CI tier-1 equivalents (docs-site build, broker pytest) were also run locally; the one broker pytest failure observed is a pre-existing flaky test bug unrelated to this change (test_bit_flipped_signature_rejectedflips the first signature character but conditions on the last).Risk
background.confirmation_required) is unchanged.Contributor Agreement
(For the author to confirm — these are personal attestations.)