fix(creds): keep the job queue alive across transient token refreshes#955
Conversation
On a transient OAuth refresh error, HttpCredentials::refreshAccessTokenInternal schedules a retry but then also emits authenticationFailed(). Account reacts to that signal with JobQueue::clear() (account.cpp), aborting every in-flight upload. The sync run then finalizes as "complete", so the aborted files are silently never retried: directories exist but files are missing, and subsequent syncs skip the already-recorded items (silent data loss). Emit authenticationFailed() only on the terminal branch (TokenRefreshMaxRetries exceeded, a real logout). On a transient error the queue stays blocked across the scheduled retry and resumes via fetched()->unblock() once the refresh succeeds. Fixes opencloud-eu#900 Refs opencloud-eu#948 Authored-By: Bernard Gütermann <bernard.gutermann@sekops.ch>
9bbc6ee to
19b0a75
Compare
Verify that transient OAuth token-refresh errors do NOT emit authenticationFailed(), which would clear the job queue and abort in-flight uploads (opencloud-eu#900, opencloud-eu#948). Also verify that after TokenRefreshMaxRetries consecutive errors, authenticationFailed() IS emitted (terminal failure). Authored-By: Bernard Gütermann <bernard.gutermann@sekops.ch>
Regression test added (f44ea97)Added two regression tests to
Prove-the-test-bites result
Full suite: 31/31 pass, 0 regressions. Built and tested in |
Make the native discover->reconcile->upload loop self-heal on recoverable errors instead of wedging or aborting the whole run (complements the token-wedge fix opencloud-eu#955): - classifyError: a transient network/timeout on one file is now a per-file NormalError + another-pass, not FatalError -> propagator()->abort() (which aborts the ENTIRE run on a single blip over a long, multi-day sync). Genuinely fatal cases (TLS handshake, proxy auth, redirects) keep FatalError. - TUS resume: a 409 Upload-Offset mismatch (opencloud-eu#898) now routes through the existing HEAD-offset-recovery path and resumes from the server's canonical offset, instead of wedging in commonErrorHandling. classifyError also maps 409 to a recoverable SoftError + another-pass. - test/testclassifyerror.cpp: regression coverage (bug-bites verified). Fixes opencloud-eu#898 Authored-By: Bernard Gütermann <bernard.gutermann@sekops.ch>
TheOneRing
left a comment
There was a problem hiding this comment.
Thanks for your contribution, if I dissect the git blame correctly that error exists for at least 3 years.
Problem
The desktop client can silently drop files when an OAuth access token is refreshed mid-sync. Afterwards directories exist on the server but files inside are missing, and later syncs skip the already-recorded items — silent, persistent loss. This is the mechanism behind #900 ("Aborted after a single 401 on
/me/drives") and #948 (a transient.well-known/openid-configurationresponse aborts all jobs).Root cause
HttpCredentials::refreshAccessTokenInternal()splits refresh errors into transient (schedule a retry) and terminal (give up afterTokenRefreshMaxRetries). The transient branch schedules the retry but then alsoQ_EMIT authenticationFailed();.Accountconnects that signal toJobQueue::clear()(src/libsync/account.cpp), whichabort()s every queued upload. Aborted-but-unsent jobs aredeleteLater()d without ever callingdone(), so no error/retry flag is set;SyncEngine::finalize()then commits the journal as "All Finished" and the dropped files are never re-discovered.So a transient refresh — which the client intends to retry — is treated as a terminal failure for the job queue.
Fix
Emit
authenticationFailed()only on the terminal branch. On a transient error the job queue stays blocked (it was blocked byauthenticationStarted()) across the scheduled retry and resumes viafetched() -> JobQueueGuard::unblock()on success. The early-returnis restructured into an explicitif (terminal) / else (transient)so the two outcomes are clear.Why it's safe
authenticationFailed()has two consumers —account.cpp(JobQueue::clear()) andcmd.cpp(qFatal()inopencloudcmd). Both should react only to a terminal failure; the CLI now waits out the bounded retry instead of dying on a transient blip. The retry is bounded byTokenRefreshMaxRetries == 3, and the terminal branch still emits the signal, so the queue cannot stay blocked forever.Reproduction
OpenCloud + per-drive OAuth (Keycloak),
mirall/3.0.3.2073, sync a folder large enough that the run outlives one access-token lifetime. At the expiry boundary the proxy logs a single401on/me/drives; the client goes "Aborted", directories are created but files are missing, and re-syncing does not recover them.Testing
I don't have a Qt build environment to run the suite here, and there is currently no test that drives
HttpCredentials::refreshAccessToken(test/testoauthcoversAccountBasedOAuth;test/testjobqueuecovers the queue primitives). The right regression test would deliver a transientrefreshErrorand assert (a)authenticationFailed()is not emitted and the queue stays blocked, then (b) afterTokenRefreshMaxRetriesit is emitted. Glad to add it if you can point me at the preferred seam (extend thetestoauthFakeAM/FakeErrorReplyharness to instantiateHttpCredentials, or an integration test intestsyncengine).Fixes #900
Refs #948