Hi, I encountered a stuck issue when training in DDP, and here is the reason I figured out:
|
pos_inds = label < self.num_classes |
|
neg_inds = label == self.num_classes |
|
if pos_inds.sum() > 0: |
|
pred_fg_distri = F.softmax(cls_score[pos_inds, :self.num_classes], dim=1) |
|
|
|
fg_confusion_matrix_tmp = torch.zeros_like(self.fg_confusion_matrix).scatter_add_(0, label[pos_inds].view(-1,1).repeat(1,self.num_classes), pred_fg_distri) |
|
|
|
fg_confusion_matrix_tmp_pool = [torch.zeros_like(fg_confusion_matrix_tmp) for i in range(torch.distributed.get_world_size())] |
|
torch.distributed.all_gather(fg_confusion_matrix_tmp_pool, fg_confusion_matrix_tmp) |
|
fg_confusion_matrix_tmp = sum(fg_confusion_matrix_tmp_pool) |
|
|
the if branch starting from line 284 may be entered for some processes, and may not for others (e.g. some training images contain no labelled instance), so the gather function in line 289 will permanently wait for those processes that never execute line 284~288.
One possible solution is to check if all processes satisfy pos_inds.sum() > 0. If not, do not update the confusion matrix.
Hi, I encountered a stuck issue when training in DDP, and here is the reason I figured out:
PCB/mmdet/models/losses/cross_entropy_pcb_loss.py
Lines 281 to 291 in 066cd92
the if branch starting from line 284 may be entered for some processes, and may not for others (e.g. some training images contain no labelled instance), so the gather function in line 289 will permanently wait for those processes that never execute line 284~288.
One possible solution is to check if all processes satisfy
pos_inds.sum() > 0. If not, do not update the confusion matrix.