How to use DDP?

[Zhiying-Li-dot]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I do not know what should I do. When I run train.py without DDP, it can run, but slowly. So I want to use DDP to speed up. Always display this bug. Could you tell me how to solve this problem? Please.

Additional

No response

[glenn-jocher]

@AyushExel it looks like we should disable LOGGER for all RANKs except for RANK in (0, -1) to avoid multiple printouts like the AMP check above:

yolov5/utils/general.py

Line 534 in 1156a32

LOGGER.warning(emojis(f'{prefix}checks failed ❌, disabling Automatic Mixed Precision. See {help_url}'))

I think I can do this here:

yolov5/utils/general.py

Lines 82 to 99 in 1156a32

	def set_logging(name=None, verbose=VERBOSE):
	# Sets level and returns logger
	if is_kaggle():
	for h in logging.root.handlers:
	logging.root.removeHandler(h) # remove all handlers associated with the root logger object
	rank = int(os.getenv('RANK', -1)) # rank in world for Multi-GPU trainings
	level = logging.INFO if verbose and rank in {-1, 0} else logging.WARNING
	log = logging.getLogger(name)
	log.setLevel(level)
	handler = logging.StreamHandler()
	handler.setFormatter(logging.Formatter("%(message)s"))
	handler.setLevel(level)
	log.addHandler(handler)
	set_logging() # run before defining LOGGER
	LOGGER = logging.getLogger("yolov5") # define globally (used in train.py, val.py, detect.py, etc.)

EDIT: Oh, haha I've already done this on L88, this line is just printing out because it's set to warning for non RANK (0, -1) messages. Maybe I should just disable it entirely or set it to logging.ERROR for non RANK(0, -1)?
EDIT2: console output fix implemented in #8284

[glenn-jocher]

@lizhiying-2019 your error doesn't look familiar. For full DDP examples please see our Multi-GPU tutorial below which recommends using our Docker image for all DDP trainings.

If you have a reproducible issue inside our Docker container please let us know with exact commands to reproduce. If we can not reproduce there is no action for us to take.

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Tips for Best Training Results  ☘️ RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Roboflow for Datasets, Labeling, and Active Learning  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• TFLite, ONNX, CoreML, TensorRT Export 🚀
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• Architecture Summary  ⭐ NEW

How to create a Minimal, Reproducible Example

When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:

• ✅ Minimal – Use as little code as possible to produce the problem
• ✅ Complete – Provide all parts someone else needs to reproduce the problem
• ✅ Reproducible – Test the code you're about to provide to make sure it reproduces the problem

For Ultralytics to provide assistance your code should also be:

• ✅ Current – Verify that your code is up-to-date with GitHub master, and if necessary git pull or git clone a new copy to ensure your problem has not already been solved in master.
• ✅ Unmodified – Your problem must be reproducible using official YOLOv5 code without changes. Ultralytics does not provide support for custom code ⚠️.

If you believe your problem meets all the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem.

Thank you! 😃