dpp training stuck ( lightning=1.4.9 )

[yinrong]

I'm stuck using lightning=1.4.9
the numbers "96% 4260/4435" keeps the same forever.

[tchaton]

Dear @yinrong,

Would it be possible for you to provide a reproducible script, possibly with the BoringModel ?

Best,
T.C

[yinrong]

Dear @yinrong,

Would it be possible for you to provide a reproducible script, possibly with the BoringModel ?

Best, T.C

@tchaton

I tested my code with fewer data ( random 1% ) and it works well ( quick runs to epoch 66 ). So I think a minimum reproducible script is hard to find.

More clues:

1. restart python, it stucks at exact the same place.
2. the training dataset is randomly picked, like pick_row() if random() < 0.5
3. when stuck, cpu/gpu are fully utilized.
   
4. ctrl+c no response. kill the process some warning shows multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown

[yinrong]

I change some code. This time it stuck at epoch-0-val 100% , and after I CTRL+C :

[yinrong]

@tchaton could you continue to work on this issue?

[tchaton]

Hey @yinrong,

Could you make sure you have the same number of batches for all ranks.

It will hang if there is an uneven number of batches

Best,
T.C

[awaelchli]

To expand on @tchaton answer: When you randomly select your dataset, make sure each process chooses the same one. You can set the seed to make sure of this:

from pytorch_lightning import seed_everything
seed_everything(1)

[yinrong]

make all gpu (also called rank) have the same number of batches fixes the issue

[colligant]

seed_everything() fixed my issue as well. I am also randomly sampling from my dataset, but I was confronted with a different traceback - pytorch_lightning.utilities.exceptions.DeadlockException and WorkNCCL(OpType=AllReduce, Timeout(ms)=1800000). Just dropping this here in case it helps anyone.

[IamHimon]

same question, have you solved it?