Commit 2edaed1
committed
CLDSRV-992: Wait for MongoDB before waiting for the S3 API
The mongodb-backed jobs start `docker compose up -d` and then immediately wait
up to 40s for port 8000. Nothing waits for MongoDB itself, and cloudserver's
start-up blocks on reaching its metadata backend, so a slow or still-starting
mongo does not surface as a mongo error: it surfaces as the S3 API never
binding 8000, and the job dies on "Server did not start in less than 40
seconds" with no indication of why.
The margin is thin even when everything is healthy. On the multiple-backend
job the API bound 8000 at t+19.4s on a passing attempt, against a 40s budget
that also has to cover image pull, container create and mongo's own replica-set
init. A runner half as fast fails, which matches the observed pattern of a job
that fails at the port wait and passes on re-run.
Gate on mongo's port first, in the four jobs that run with S3METADATA=mongodb
and previously waited only on 8000: multiple-backend, mongo-v0-ft-tests,
mongo-v1-ft-tests and sur-tests. This is the same dependency-ordering gap as
the file daemons, and the same remedy: wait on the observable condition rather
than hoping the API's own wait absorbs it. A healthy run pays nothing, since
the wait returns as soon as the port answers.
The failing attempt's own server log is no longer available to confirm this
directly: the artifacts bundle is per run, so the successful re-run overwrote
it. What the surviving log does establish is the 19.4s bind and that mongo is
un-gated. Treat this as the probable cause rather than a proven one; if the
failure recurs after this change, the next occurrence will at least fail at the
mongo wait and name the real culprit.
Clears CLDSRV-992 row F5 (run 33639273186).
Issue: CLDSRV-9921 parent 4628dd2 commit 2edaed1
1 file changed
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
336 | 336 | | |
337 | 337 | | |
338 | 338 | | |
| 339 | + | |
| 340 | + | |
339 | 341 | | |
340 | 342 | | |
341 | 343 | | |
| |||
398 | 400 | | |
399 | 401 | | |
400 | 402 | | |
| 403 | + | |
| 404 | + | |
401 | 405 | | |
402 | 406 | | |
403 | 407 | | |
| |||
459 | 463 | | |
460 | 464 | | |
461 | 465 | | |
| 466 | + | |
| 467 | + | |
462 | 468 | | |
463 | 469 | | |
464 | 470 | | |
| |||
771 | 777 | | |
772 | 778 | | |
773 | 779 | | |
| 780 | + | |
| 781 | + | |
774 | 782 | | |
775 | 783 | | |
776 | 784 | | |
| |||
0 commit comments