CLDSRV-992: Wait for the file data daemon before the S3 API

`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.

Context on why cloudserver does not catch this itself. metadata.setup() opens
the metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata, which matches the run: CreateBucket and every metadata
read succeeded and only the data path failed. Nothing guards the data daemon,
because clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients
that implement no probe and DataFileInterface implements none. So the deep
healthcheck only really checks metadata, and only for the mongo and bucketd
clients. Adding the file backend to the deep check would let the product gate
itself and make this change unnecessary, but that belongs in Arsenal.

Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.

Clears CLDSRV-992 row F1 (ten of its twelve failures) and is the leading
candidate for rows F2 and F3, whose jobs share this boot path.

Issue: CLDSRV-992

Author: anurag4DSB

.github/workflows/tests.yaml

@@ -532,6 +532,8 @@ jobs:
       - name: Run file ft tests
         run: |-
           set -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           yarn run ft_test | tee /tmp/artifacts/${{ matrix.job-name }}/tests.log
       - name: Cleanup and upload coverage
@@ -831,6 +833,8 @@ jobs:
       - name: Run file KMIP tests
         run: |-
           set -ex -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           bash wait_for_local_port.bash 5696 40
           yarn run ft_kmip | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -909,6 +913,8 @@ jobs:
           S3_END_TO_END: true # to use the default credentials profile and not vault profile
         run: |-
           set -ex -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           bash wait_for_local_port.bash 5696 40
           yarn run ft_kmip_cluster | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -1009,6 +1015,8 @@ jobs:
       - name: Wait for services vault and s3
         run: |-
           bash wait_for_local_port.bash 8500 40
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
       - name: Ensure old version of cloudserver and vault is used
         run: |-
@@ -1053,6 +1061,8 @@ jobs:
         run: |-
           bash wait_for_local_port.bash ${{ matrix.kms.port }} 40
           bash wait_for_local_port.bash 8500 40
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
       - name: Ensure latest version of cloudserver and vault is used
         run: |-