CLDSRV-992: Wait for a mongo primary before waiting for the S3 API

The mongodb-backed jobs start `docker compose up -d` and then immediately wait
up to 40s for port 8000. Nothing waits for MongoDB, and cloudserver's start-up
blocks on reaching its metadata backend, so a slow or not-yet-elected mongo does
not surface as a mongo error: it surfaces as the S3 API never binding 8000, and
the job dies on "Server did not start in less than 40 seconds" with no
indication of why.

A TCP check on 27018 would not be enough. mongo-run.sh starts mongod straight
away but backgrounds rs.initiate() behind a `sleep 5`, so the port answers at
least five seconds before the replica set exists. Gate on an elected primary
instead -- rs.status().myState == 1 -- which is the condition cloudserver
actually needs.

The margin is thin even when everything is healthy. On the multiple-backend job
the API bound 8000 at t+19.4s on a passing attempt, against a 40s budget that
also has to cover image pull, container create and the replica-set init above.
A runner half as fast fails, which matches the observed pattern of a job that
fails at the port wait and passes on re-run.

Applied to the four jobs that run with S3METADATA=mongodb and previously waited
only on 8000: multiple-backend, mongo-v0-ft-tests, mongo-v1-ft-tests and
sur-tests. A healthy run pays nothing, since the wait returns as soon as a
primary is reported. On timeout it dumps rs.status(), so the next occurrence
names the real culprit instead of blaming the S3 API.

The failing attempt's own server log is no longer available to confirm this
directly: the artifacts bundle is per run, so the successful re-run overwrote
it. What the surviving log does establish is the 19.4s bind and that mongo is
un-gated. Treat this as the probable cause rather than a proven one.

Clears CLDSRV-992 row F5 (run 33639273186).

Issue: CLDSRV-992

Author: anurag4DSB

.github/workflows/tests.yaml

@@ -336,6 +336,8 @@ jobs:
       - name: Run multiple backend test
         run: |-
           set -o pipefail;
+          # start-up blocks on mongo, so a slow mongo looks like 8000 never binding
+          bash wait_for_mongo_primary.bash 120
           bash wait_for_local_port.bash 8000 40
           bash wait_for_local_port.bash 81 40
           yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -398,6 +400,8 @@ jobs:
       - name: Run functional tests
         run: |-
           set -o pipefail;
+          # start-up blocks on mongo, so a slow mongo looks like 8000 never binding
+          bash wait_for_mongo_primary.bash 120
           bash wait_for_local_port.bash 8000 40
           yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
         env:
@@ -459,6 +463,8 @@ jobs:
       - name: Run functional tests
         run: |-
           set -o pipefail;
+          # start-up blocks on mongo, so a slow mongo looks like 8000 never binding
+          bash wait_for_mongo_primary.bash 120
           bash wait_for_local_port.bash 8000 40
           yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
           yarn run ft_mixed_bucket_format_version | tee /tmp/artifacts/${{ github.job }}/mixed-tests.log
@@ -771,6 +777,8 @@ jobs:
       - name: Run SUR-related tests
         run: |-
           set -ex -o pipefail;
+          # start-up blocks on mongo, so a slow mongo looks like 8000 never binding
+          bash wait_for_mongo_primary.bash 120
           bash wait_for_local_port.bash 8000 40
           yarn run test_sur | tee /tmp/artifacts/${{ github.job }}/tests.log
       - name: Cleanup and upload coverage

wait_for_mongo_primary.bash

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Wait until the mongo replica set has elected a primary.
+#
+# mongo-run.sh starts mongod immediately but backgrounds rs.initiate() behind a
+# `sleep 5`, so the port answers well before the set exists. A TCP check
+# therefore lets tests start against a mongo with no primary, and cloudserver,
+# which blocks on its metadata backend during start-up, then never binds 8000.
+#
+# usage: wait_for_mongo_primary.bash [timeout_seconds]
+set -uo pipefail
+
+timeout=${1:-120}
+count=0
+
+# compose was brought up from .github/docker, so resolve the project from there
+cd "$(dirname "$0")/.github/docker" || exit 1
+
+mongo_state() {
+    docker compose exec -T mongo \
+        mongo --port 27018 --quiet --eval 'rs.status().myState' 2>/dev/null \
+        | tr -dc '0-9'
+}
+
+echo "waiting for mongo replica set primary"
+while [ "$count" -lt "$timeout" ]; do
+    # myState 1 is PRIMARY; anything else, or a failed exec, means not ready
+    if [ "$(mongo_state)" = "1" ]; then
+        echo ""
+        echo "Mongo primary ready in ~${count} seconds. Starting test now..."
+        exit 0
+    fi
+    echo -n .
+    sleep 1
+    count=$((count + 1))
+done
+
+echo ""
+echo "Mongo replica set had no primary after ${timeout} seconds. Exiting..."
+docker compose exec -T mongo \
+    mongo --port 27018 --quiet --eval 'rs.status()' 2>&1 | tail -30 || true
+exit 1