Do training params impact inference speed for the same model size?

[DSA101]

I produced several versions of a model custom trained from yolov5m checkpoint, with the same 640 image size. The model trained with 1K images seems to be twice as fast as the model trained on 3K images. Is this expected or am I missing something? I would imagine that the inference speed should be the same, since the model size remains the same (I checked from logs that it is the case). I noticed however that the faster model is 55MB in size, while the slower one is 42MB.

Sorry for the noob questions and thanks for any ideas.

[glenn-jocher]

@DSA101 👋 Hello! Thanks for asking about inference speed issues. YOLOv5 🚀 can be run on CPU (i.e. --device cpu, slow) or GPU if available (i.e. --device 0, faster). You can determine your inference device by viewing the YOLOv5 console output:

detect.py inference

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/

YOLOv5 PyTorch Hub inference

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()
# Speed: 631.5ms pre-process, 19.2ms inference, 1.6ms NMS per image at shape (2, 3, 640, 640)

Increase Speeds

If you would like to increase your inference speed some options are:

• Use batched inference with YOLOv5 PyTorch Hub
• Reduce --img-size, i.e. 1280 -> 640 -> 320
• Reduce model size, i.e. YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s -> YOLOv5n
• Use half precision FP16 inference with python detect.py --half and python val.py --half
• Use a faster GPUs, i.e.: P100 -> V100 -> A100
• Export to ONNX for up to 3x CPU speedup
• Export to TensorRT for up to 5x GPU speedup
• Use a free GPU backends with up to 16GB of CUDA memory:

Good luck 🍀 and let us know if you have any other questions!

[DSA101]

That's all reasonable suggestions, but that's not an answer to my question. My question is this: can two different versions of the same model (trained with different number of images, different batch size, but the same model size) result in significantly different (2X in my case) inference speed? Or the inference speed of the same model size (e.g. a model trained based on yolov5m) is expected always to be the same irrespective of training data since the number of parameters is the same?

I just want to understand whether it's not unexpected that the model trained on 3K images is slower than the model trained on 1K images on the same system with GPU inference working in both cases, or it is unexpected and thus I need to debug what may be causing the inference speed difference.

[glenn-jocher]

@DSA101 identical model architectures naturally produce identical inference speeds.

[DSA101]

Thank you! That's what I wanted to confirm.