CLDSRV-992: Capture cloudserver's log on the s3c functional jobs - #6283
Draft
anurag4DSB wants to merge 1 commit into
Draft
CLDSRV-992: Capture cloudserver's log on the s3c functional jobs#6283anurag4DSB wants to merge 1 commit into
anurag4DSB wants to merge 1 commit into
Conversation
The s3c-ft-tests jobs run cloudserver as a container and upload sproxyd, bucketd, repd and vault logs, but never cloudserver's own. Its output goes to the container's stdout and is discarded, so any failure that originates in cloudserver on these jobs is undiagnosable after the fact. That is not hypothetical. Three CLDSRV-992 rows sit on these jobs and are all stuck for exactly this reason: - F8's ServiceUnavailable on "should get the body of part 8 when ordered MPU". Chasing a fresh occurrence showed nginx with zero 5xx, sproxyd with zero errors, and bucketd with zero 503s and its only connection errors at a single startup instant. The 503 was therefore generated by cloudserver itself, and there is no log to say why. - F4 and F8's 40s timeouts in the uploadPartCopy fixtures. The same fixture builds successfully three or four times per run and then stalls once, so it is an intermittent stall rather than slow work, and again nothing records it. Dump the container's log to the job's artifacts directory before teardown, while the containers still exist. The step is if: always() and ends in `|| true`, so it runs on failure -- which is when it matters -- and can never itself fail a job. A passing run pays one docker compose logs call. Issue: CLDSRV-992
This was referenced Sep 8, 2026
Contributor
Hello anurag4dsb,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
@@ Coverage Diff @@
## development/9.3 #6283 +/- ##
===================================================
- Coverage 85.31% 85.28% -0.03%
===================================================
Files 206 206
Lines 13435 13435
===================================================
- Hits 11462 11458 -4
- Misses 1973 1977 +4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
anurag4DSB
marked this pull request as draft
September 8, 2026 13:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The s3c-ft-tests jobs upload sproxyd, bucketd, repd and vault logs but never cloudserver's own, because its output goes to the container's stdout and is discarded. Any failure originating in cloudserver on these jobs is therefore undiagnosable after the fact, and that is currently blocking three CLDSRV-992 rows rather than being hypothetical.
Chasing a fresh occurrence of F8's ServiceUnavailable through the artifacts that do exist: nginx logged zero 5xx, sproxyd zero errors, and bucketd zero 503s with its only connection errors at a single startup instant. So cloudserver produced that 503 itself and nothing recorded why. Same for F4's and F8's 40s fixture timeouts, where the same fixture builds three or four times per run and stalls once, which is a stall rather than slow work.
The job now dumps cloudserver's container log to its artifacts directory before teardown, while the containers still exist. The step is
if: always()and ends in|| true, so it runs on failure, which is when it matters, and can never itself fail a job; a passing run pays onedocker compose logscall. If it were wrong we would see an empty or missing s3.log in the very next run's artifacts.