CLDSRV-992: Bound and settle the 1000-object writes in Multi-Object Delete setup - #6278
Draft
anurag4DSB wants to merge 1 commit into
Draft
Conversation
Contributor
Hello anurag4dsb,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
❌ 1 Tests Failed:
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| async-migration-report: | ||
| runs-on: ubuntu-24.04 | ||
| if: startsWith(github.ref, 'refs/heads/development/') | ||
| if: false # CLDSRV-992 stress: unrelated job |
There was a problem hiding this comment.
This file contains stress-testing scaffolding that should be reverted before merge:
- 11 jobs disabled with
if: false(async-migration-report, multiple-backend, mongo-v0, mongo-v1, file-ft, utapi-v2, sur, kmip, kmip-cluster, sse-kms-migration, end-of-pipeline aggregation). Merging this removes most of the CI safety net ondevelopment/9.3. - s3c-ft-tests matrix drops the
v0andv0-null-compatvariants and duplicatesv110 times. v0 format test coverage would be lost. - The end-of-pipeline aggregation job changes from
if: always()toif: false, silently dropping the merge gate.
Only the multiObjectDelete.js change should ship; the workflow changes should be reverted to their base-branch state.
…elete setup The "Multi-Object Delete Success" beforeEach uploads 1000 objects behind what looks like a concurrency limiter of 20. It is not one. objects.map() invokes all 1000 async calls synchronously; the first 20 push and the other 980 all park on the same `await Promise.race(queued)`. When the first put settles they all wake, and each splices one entry off the head of the queue regardless of whether that entry had settled, so the queue empties, every waiter's `while` test goes false, and the remaining puts fire together. Running the limiter against an instrumented stub gives a peak of 999 concurrent puts against an intended 20. That is the ServiceUnavailable and the "Socket timed out without establishing a connection within 5000 ms" seen in CI: a thousand simultaneous puts, not twenty. It also explains why one failure becomes seven. Promise.all rejects on the first error while the other puts are still in flight, so the hook throws with writes outstanding. Teardown then empties the bucket, the stragglers land, and deleteBucket fails with BucketNotEmpty. All four suites in this file share the bucket name multi-object-delete-234-634, so once it is left behind with objects in it, every later suite's cleanup fails the same way. Replace the limiter with a fixed pool of 20 workers draining a shared queue. Concurrency is genuinely bounded, and every put is settled before the hook returns, on the failure path too, so teardown can no longer race them. Errors are collected and the first is rethrown, which keeps the diagnostic the hook previously reported. Clears CLDSRV-992 rows F1 (its two Multi-Object Delete hooks, run 33734792029) and F10 (all seven failures, run 33681270087, fixed for the 9.2 line separately). Issue: CLDSRV-992
anurag4DSB
force-pushed
the
improvement/CLDSRV-992-multiobjectdelete-bounded-writes
branch
2 times, most recently
from
September 8, 2026 11:02
57f5b07 to
9d47a15
Compare
anurag4DSB
marked this pull request as ready for review
September 8, 2026 11:02
anurag4DSB
marked this pull request as draft
September 8, 2026 13:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
Multi-Object Delete Successsetup uploads 1000 objects behind what looks like a concurrency limiter of 20, and it is not one.objects.map()invokes all 1000 async calls synchronously; the first 20 push and the other 980 park on the samePromise.race, then on the first settle each waiter splices one entry off the head of the queue regardless of whether that entry had settled, so the queue empties and the remaining puts all fire together. Lifting the limiter verbatim into a harness with an instrumented stub gives a peak of 999 concurrent puts against an intended 20, which is where the ServiceUnavailable and theSocket timed out without establishing a connection within 5000 mscome from.It also explains why one failure becomes seven.
Promise.allrejects on the first error while the other puts are still in flight, so the hook throws with writes outstanding; teardown then empties the bucket, the stragglers land, anddeleteBucketfails with BucketNotEmpty. All four suites in the file share the bucket namemulti-object-delete-234-634, so once it is left behind non-empty every later suite's cleanup fails the same way.The limiter becomes a fixed pool of 20 workers draining a shared queue, so concurrency is genuinely bounded and every put has settled before the hook returns, including on the failure path. Errors are collected and the first is rethrown, so the diagnostic CI reports is unchanged. The replacement measures a peak of exactly 20 with nothing in flight at return, on the happy path and with an injected mid-run failure. Clears row F1's two Multi-Object Delete hooks; hotfix/9.2.36 needs this plus a cleanup change it never received, in #6279.