Skip to content

Commit 2edaed1

Browse files
committed
CLDSRV-992: Wait for MongoDB before waiting for the S3 API
The mongodb-backed jobs start `docker compose up -d` and then immediately wait up to 40s for port 8000. Nothing waits for MongoDB itself, and cloudserver's start-up blocks on reaching its metadata backend, so a slow or still-starting mongo does not surface as a mongo error: it surfaces as the S3 API never binding 8000, and the job dies on "Server did not start in less than 40 seconds" with no indication of why. The margin is thin even when everything is healthy. On the multiple-backend job the API bound 8000 at t+19.4s on a passing attempt, against a 40s budget that also has to cover image pull, container create and mongo's own replica-set init. A runner half as fast fails, which matches the observed pattern of a job that fails at the port wait and passes on re-run. Gate on mongo's port first, in the four jobs that run with S3METADATA=mongodb and previously waited only on 8000: multiple-backend, mongo-v0-ft-tests, mongo-v1-ft-tests and sur-tests. This is the same dependency-ordering gap as the file daemons, and the same remedy: wait on the observable condition rather than hoping the API's own wait absorbs it. A healthy run pays nothing, since the wait returns as soon as the port answers. The failing attempt's own server log is no longer available to confirm this directly: the artifacts bundle is per run, so the successful re-run overwrote it. What the surviving log does establish is the 19.4s bind and that mongo is un-gated. Treat this as the probable cause rather than a proven one; if the failure recurs after this change, the next occurrence will at least fail at the mongo wait and name the real culprit. Clears CLDSRV-992 row F5 (run 33639273186). Issue: CLDSRV-992
1 parent 4628dd2 commit 2edaed1

1 file changed

Lines changed: 8 additions & 0 deletions

File tree

.github/workflows/tests.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,8 @@ jobs:
336336
- name: Run multiple backend test
337337
run: |-
338338
set -o pipefail;
339+
# start-up blocks on mongo, so a slow mongo looks like 8000 never binding
340+
bash wait_for_local_port.bash 27018 120
339341
bash wait_for_local_port.bash 8000 40
340342
bash wait_for_local_port.bash 81 40
341343
yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/tests.log
@@ -398,6 +400,8 @@ jobs:
398400
- name: Run functional tests
399401
run: |-
400402
set -o pipefail;
403+
# start-up blocks on mongo, so a slow mongo looks like 8000 never binding
404+
bash wait_for_local_port.bash 27018 120
401405
bash wait_for_local_port.bash 8000 40
402406
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
403407
env:
@@ -459,6 +463,8 @@ jobs:
459463
- name: Run functional tests
460464
run: |-
461465
set -o pipefail;
466+
# start-up blocks on mongo, so a slow mongo looks like 8000 never binding
467+
bash wait_for_local_port.bash 27018 120
462468
bash wait_for_local_port.bash 8000 40
463469
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
464470
yarn run ft_mixed_bucket_format_version | tee /tmp/artifacts/${{ github.job }}/mixed-tests.log
@@ -771,6 +777,8 @@ jobs:
771777
- name: Run SUR-related tests
772778
run: |-
773779
set -ex -o pipefail;
780+
# start-up blocks on mongo, so a slow mongo looks like 8000 never binding
781+
bash wait_for_local_port.bash 27018 120
774782
bash wait_for_local_port.bash 8000 40
775783
yarn run test_sur | tee /tmp/artifacts/${{ github.job }}/tests.log
776784
- name: Cleanup and upload coverage

0 commit comments

Comments
 (0)