CLDSRV-992: Bound and settle the 1000-object writes in Multi-Object Delete setup

The "Multi-Object Delete Success" beforeEach uploads 1000 objects behind what
looks like a concurrency limiter of 20. It is not one. objects.map() invokes all
1000 async calls synchronously; the first 20 push and the other 980 all park on
the same `await Promise.race(queued)`. When the first put settles they all wake,
and each splices one entry off the head of the queue regardless of whether that
entry had settled, so the queue empties, every waiter's `while` test goes false,
and the remaining puts fire together. Running the limiter against an
instrumented stub gives a peak of 999 concurrent puts against an intended 20.

That is the ServiceUnavailable seen in run 33681270087: a thousand simultaneous
puts, not twenty.

It also explains why one failure becomes seven. Promise.all rejects on the first
error while the other puts are still in flight, so the hook throws with writes
outstanding, and the bucket cannot be deleted. All four suites in this file
share the bucket name multi-object-delete-234-634, so once it is left behind
with objects in it, every later suite's cleanup fails with BucketNotEmpty --
which is exactly the shape of the run: one ServiceUnavailable followed by six
BucketNotEmpty.

Two changes, both already present on development/9.3 and newer in one form or
another:

- Replace the limiter with a fixed pool of 20 workers draining a shared queue.
  Concurrency is genuinely bounded and every put is settled before the hook
  returns, on the failure path too, so teardown can no longer race them.
- Empty the bucket before deleting it in all four cleanup hooks. The 9.3 line
  gained this in 9eb5777, but that commit is an aws-sdk v3 migration of the
  whole file and is not something to cherry-pick onto 9.2; this is the
  equivalent written for the v2 client, and nothing else from it.

Clears CLDSRV-992 row F10 (all seven failures).

Issue: CLDSRV-992

Author: anurag4DSB

tests/functional/aws-node-sdk/test/object/multiObjectDelete.js

@@ -68,29 +68,43 @@ describe('Multi-Object Delete Success', function success() {
                 objects.push(`${key}${i}`);
             }
             const parallel = 20;
-            const queued = [];
-            const putObjectWithLimit = async key => {
-                while (queued.length >= parallel) {
-                    await Promise.race(queued);
-                    queued.splice(0, queued.findIndex(p => p === queued[0]) + 1);
+            // A fixed pool of workers draining a shared queue. The previous
+            // limiter admitted every put at once (measured peak in flight: 999
+            // of 1000), which is what provoked the ServiceUnavailable; and
+            // because Promise.all rejects on the first failure, it left the
+            // rest running, so writes kept landing after the teardown had
+            // emptied the bucket and deleteBucket then failed with
+            // BucketNotEmpty. Here concurrency is really bounded, and every put
+            // has settled before this hook returns, failure included.
+            const pending = objects.slice();
+            const putErrors = [];
+            await Promise.all(Array.from({ length: parallel }, async () => {
+                while (pending.length > 0) {
+                    const objectKey = pending.shift();
+                    try {
+                        await s3.putObject({
+                            Bucket: bucketName,
+                            Key: objectKey,
+                            Body: 'somebody',
+                        }).promise();
+                    } catch (err) {
+                        putErrors.push(err);
+                    }
                 }
-                const result = s3.putObject({
-                    Bucket: bucketName,
-                    Key: key,
-                    Body: 'somebody',
-                }).promise();
-                queued.push(result);
-                return result;
-            };
-            const putPromises = objects.map(key => putObjectWithLimit(key));
-            await Promise.all(putPromises);
+            }));
+            if (putErrors.length > 0) {
+                throw putErrors[0];
+            }
         } catch (err) {
             process.stdout.write(`Error creating objects: ${err}\n`);
             throw err;
         }
     });
 
-    afterEach(() => s3.deleteBucket({ Bucket: bucketName }).promise());
+    afterEach(async () => {
+        await bucketUtil.empty(bucketName);
+        await s3.deleteBucket({ Bucket: bucketName }).promise();
+    });
 
     it('should batch delete 1000 objects', done => {
         const objects = createObjectsList(1000);
@@ -150,7 +164,10 @@ describe('Multi-Object Delete Error Responses', () => {
             });
         });
 
-        afterEach(() => s3.deleteBucket({ Bucket: bucketName }).promise());
+        afterEach(async () => {
+            await bucketUtil.empty(bucketName);
+            await s3.deleteBucket({ Bucket: bucketName }).promise();
+        });
 
         it('should return error if request deletion of more than 1000 objects',
             () => {
@@ -240,7 +257,10 @@ describe('Multi-Object Delete Access', function access() {
         });
     });
 
-    after(() => s3.deleteBucket({ Bucket: bucketName }).promise());
+    after(async () => {
+        await bucketUtil.empty(bucketName);
+        await s3.deleteBucket({ Bucket: bucketName }).promise();
+    });
 
     it('should return access denied error for each object where no acl ' +
         'permission', () => {
@@ -339,7 +359,10 @@ describeSkipIfCeph('Multi-Object Delete with Object Lock', () => {
         });
     });
 
-    after(() => s3.deleteBucket({ Bucket: bucketName }).promise());
+    after(async () => {
+        await bucketUtil.empty(bucketName);
+        await s3.deleteBucket({ Bucket: bucketName }).promise();
+    });
 
     it('should not delete locked objects', () => {
         const objects = createObjectsList(5, versionIds);