CLDSRV-992: Wait for the file data daemon before the S3 API

`yarn start` launches the S3 API and the file daemons as separate parallel
processes (npm-run-all --parallel start_dmd start_s3server), so they race. On
run 33734792029 the API bound :8000 at t+10s and served writes from t+15s while
the dataserver only bound :9991 at t+84s, and the ten HTTP 500s in between are
all ECONNREFUSED 0.0.0.0:9991. The same subdir pre-creation step took 2.7s on a
healthy runner, which is why this is intermittent.

Cloudserver's own startup partly guards this already: metadata.setup() opens the
metadata client and throws if 9990 is unreachable, so the API cannot bind
without working metadata. This branch's Arsenal pin was checked and behaves the
same way, so only the data daemon needs a gate. Nothing guards that, because
clientCheck returns a hardcoded { code: 200, message: 'OK' } for clients that
implement no probe and DataFileInterface implements none, so the deep
healthcheck only really checks metadata and only for the mongo and bucketd
clients.

Gate on 9991 in the four jobs that use the file data backend: file-ft-tests,
kmip-ft-tests, kmip-cluster-ft-tests and sse-kms-migration-tests, the last of
which starts a fresh cloudserver container in a second phase and needs the gate
there too. 120s because 84s was observed; a healthy run pays nothing since the
wait returns as soon as the port answers.

Backport of #6277

Issue: CLDSRV-992

Author: anurag4DSB

.github/workflows/tests.yaml

@@ -509,6 +509,8 @@ jobs:
       - name: Run file ft tests
         run: |-
           set -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           yarn run ft_test | tee /tmp/artifacts/${{ matrix.job-name }}/tests.log
       - name: Cleanup and upload coverage
@@ -808,6 +810,8 @@ jobs:
       - name: Run file KMIP tests
         run: |-
           set -ex -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           bash wait_for_local_port.bash 5696 40
           yarn run ft_kmip | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -883,6 +887,8 @@ jobs:
           S3_END_TO_END: true # to use the default credentials profile and not vault profile
         run: |-
           set -ex -o pipefail;
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
           bash wait_for_local_port.bash 5696 40
           yarn run ft_kmip_cluster | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -983,6 +989,8 @@ jobs:
       - name: Wait for services vault and s3
         run: |-
           bash wait_for_local_port.bash 8500 40
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
       - name: Ensure old version of cloudserver and vault is used
         run: |-
@@ -1027,6 +1035,8 @@ jobs:
         run: |-
           bash wait_for_local_port.bash ${{ matrix.kms.port }} 40
           bash wait_for_local_port.bash 8500 40
+          # the data daemon binds after 8000 and nothing in startup waits for it
+          bash wait_for_local_port.bash 9991 120
           bash wait_for_local_port.bash 8000 40
       - name: Ensure latest version of cloudserver and vault is used
         run: |-