NCCL PG timeout 30min -> 10min - #8253
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb3fab4f7a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # To make an attempt at backwards compatibility with THD, we use an | ||
| # extraordinarily high default timeout, given that THD did not have timeouts. | ||
| default_pg_timeout = timedelta(minutes=int(os.getenv("DEEPSPEED_TIMEOUT", default=30))) | ||
| default_pg_timeout = timedelta(minutes=int(os.getenv("DEEPSPEED_TIMEOUT", default=10))) |
There was a problem hiding this comment.
This single default_pg_timeout is used by init_distributed() for every backend, including explicit dist_backend="gloo" and mixed strings like cpu:gloo,cuda:nccl, before being passed directly to torch.distributed.init_process_group. Lowering it globally to 10 minutes matches the NCCL default but also cuts Gloo/CPU jobs from the previous 30-minute window, so any CPU or mixed-backend DeepSpeed run with a legitimate Gloo collective between 10 and 30 minutes will now abort unexpectedly. Consider making the default backend-specific instead of changing the shared constant.
Useful? React with 👍 / 👎.
2 participants
Apparently nccl/pytorch switched to a 10min collective timeout (from the original 30min) in 2024 (torch==2.2) but ds is at 30min still - should we sync with the more realistic timeout? 30min is a way too long, no collective takes more than 10min these days