Skip to content

Fix of Model HTTP retry/backoff + real background-job cancellation#129

Open
anon13377331 wants to merge 3 commits into
secondly-com:mainfrom
anon13377331:main
Open

Fix of Model HTTP retry/backoff + real background-job cancellation#129
anon13377331 wants to merge 3 commits into
secondly-com:mainfrom
anon13377331:main

Conversation

@anon13377331

Copy link
Copy Markdown

Summary

Fixes #88 (Model HTTP retry/backoff + real background-job cancellation).

  • Bounded retry with backoff for model HTTP calls. OpenAiResponsesAgentAdapter now funnels all Responses API calls through a single postResponses path (removing three duplicated HTTP blocks) that retries 429/5xx up to 3 attempts with exponential backoff plus jitter, honors Retry-After (both delta-seconds and HTTP-date forms), and fails fast when Retry-After exceeds the 30s in-call budget. Backoff waits poll for cancellation on a monotonic clock, and non-transient statuses (e.g. 400) are not retried. Timeouts and error-message format are unchanged.
  • Real background-job cancellation. OpenPhoneAgentJobScheduler.isCancelled() no longer hardcodes false: each run gets a JobCancellationSignal backed by the job store, keyed by a run-ownership token (the markRunning timestamp). background_job_stop (or stuck-run repair) makes the signal fire; a watchdog thread then repeatedly calls adapter.cancel() so even a thread blocked on a stuck upstream read is torn down. All AgentJobStore read-modify-write cycles are serialized on a process-wide lock (callers construct independent instances over one SharedPreferences file, so instance-level synchronized provided no exclusion), and markCompleted/markFailed/markDispatched refuse to record for a run that is no longer live — a stopped job can't be resurrected or notify as finished.
  • RuntimeException-safe job path. The job thread wraps the entire run; a broker outage or other unchecked exception records a failed run with the store's failure backoff instead of killing the assistant process. Runs ending in infrastructure errors (network_error, adapter_error) now follow failure backoff and repeated-failure alerting rather than counting as completions. scheduleNext keeps the alarm armed while any job is marked running so a run stranded by process death is repaired without waiting for a boot.
  • Docs: docs/AGENT_RUNTIME_V1.md gains a "Cancellation and Upstream Failures" section; changelog updated under Unreleased.

Validation

  • ./scripts/check.sh — passes, including the assistant javac gate (86 files against android-36) and the broker smoke test
  • git diff --check — clean
  • Device test, if relevant: not run — no ADB-connected device or emulator is available in the authoring environment. Requesting a maintainer-triggered run-gcp-lab emulator run for on-device validation (this change affects background-job behavior).
  • Screenshot, trajectory, audit export, or release artifact evidence, if relevant: behavioral evidence from a local JVM harness driving the real adapter against a stub HTTP server — 15 assertions covering 429→success retry with jittered backoff, Retry-After preference, fail-fast on Retry-After beyond the budget, bounded attempts on persistent 500, no retry on 400, and cancel-during-backoff stopping within ~1s and producing the "stopped" reply. CI tier-1 equivalents (docs-site build, broker pytest) were also run locally; the one broker pytest failure observed is a pre-existing flaky test bug unrelated to this change (test_bit_flipped_signature_rejected flips the first signature character but conditions on the last).

Risk

  • No API keys, private SSH keys, signing keys, generated build artifacts, personal device data, or proprietary vendor files are included.
  • Licensing impact reviewed for any new third-party material — none added.
  • Privacy/security impact reviewed for assistant, watcher, background-run, audit, model-provider, or device-control changes: no new data leaves the device; the change only adds retry timing, cancellation checks, and failure accounting around existing calls. The background-tool confirmation gate (background.confirmation_required) is unchanged.
  • Docs updated for user-visible behavior, build flow, or release process changes.

Contributor Agreement

(For the author to confirm — these are personal attestations.)

  • I have read and agree to CONTRIBUTING.md, including the contributor assignment/license/no-compensation terms.
  • I have the right to submit this contribution under those terms.
  • This contribution does not include material I am not allowed to submit.

claude and others added 3 commits July 8, 2026 04:58
OpenAiResponsesAgentAdapter now routes every Responses call through one
HTTP path that retries 429/5xx with bounded exponential backoff plus
jitter, honors Retry-After (seconds or HTTP-date), and polls for
cancellation while backing off. runTask also catches RuntimeException so
a misbehaving upstream surfaces as a failed task result instead of an
uncaught exception.

OpenPhoneAgentJobScheduler wires ToolExecutor.isCancelled() to the job
store: a run counts as cancelled once its stored status leaves
"running" (background_job_stop or stuck-run repair), and a watchdog
thread hard-cancels the adapter so a stopped job stops even while
blocked on a stuck upstream read. Job runner threads catch
RuntimeException and record a failed run with backoff instead of
crashing the assistant process.

AgentJobStore gains statusOf() and live-run guards on markCompleted and
markFailed so a finishing or failing run cannot resurrect a job the
user already stopped; notifications follow the recorded outcome.

Validation: scripts/check.sh passes (including check-assistant-java
against android-36); a local JVM harness drove the real adapter against
a stub HTTP server covering 429-then-success, Retry-After preference,
bounded attempts on persistent 500, no retry on 400, and cancel during
backoff. No device eval was run (no device available in this
environment).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASVb6mqLiGd7naEnLaoKnX
Serialize all AgentJobStore mutations on a process-wide lock: every
caller constructs its own store over the same SharedPreferences file, so
the previous instance-level synchronized let a concurrent
background_job_stop be erased by a job thread committing a stale jobs
array (read-modify-write race). markCompleted/markFailed/markDispatched
now also require a run-ownership token (the markRunning timestamp) so a
zombie run reclaimed by stuck-run repair cannot record over its
successor, and a stop that lands during runtime dispatch wins.

The cancel watchdog now keeps re-cancelling every poll until the run
finishes instead of exiting after the first cancel(), and the adapter
checks cancellation before opening a request and again after publishing
the connection, closing the window where a cancel saw no connection to
disconnect and the following stuck read had nothing to break it.

Background runs that end in an infrastructure error (network_error from
exhausted retries, adapter_error from an unchecked crash) are now
recorded as failures with the job store's backoff and repeated-failure
alerting instead of counting as completed runs. A Retry-After longer
than the in-call retry budget fails the call immediately rather than
retrying inside the server's declared closed window. Backoff deadlines
and cancellation-poll throttling use the monotonic clock, and
scheduleNext keeps the alarm armed while any job is marked running so a
run stranded by process death is repaired without waiting for a boot.

Validation: scripts/check.sh passes (javac gate against android-36);
the local JVM harness now also covers fail-fast on Retry-After beyond
the retry budget, alongside 429-then-success, Retry-After preference,
bounded attempts on persistent 500, no retry on 400, and cancel during
backoff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASVb6mqLiGd7naEnLaoKnX
…ation-91oqc5

Model HTTP retry/backoff and real background-job cancellation
@vercel

vercel Bot commented Jul 9, 2026

Copy link
Copy Markdown

@anon13377331 is attempting to deploy a commit to the adam-7512's projects Team on Vercel.

A member of the Team first needs to authorize it.

@anon13377331

Copy link
Copy Markdown
Author

please tell me if there is anything else I can help with on my side (currently waiting for approval of the workflow run-gcp-lab)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model HTTP retry/backoff + real background-job cancellation

2 participants