How to combine two models: one from yolov3 and another from yolov5?

[ifahim]

❔Question

I have trained a model using Ultralytics yolov3 (#classes =1, yolov3_best.pt) and pretty satisfied with that. I have another model that I trained on Ultralytics yolov5x (#classes =1, yolov5_best.pt). I am planning to create one torchscript by merging these two models.

Is there a way I can combine these two models trained on yolov3 and v5? Can anyone point to the example script that does this?

Additional context

[glenn-jocher]

@ifahim if you have multiple models trained on the same classes, you can ensemble them for improved results by passing them to detect.py together:

python detect.py --weights model1.pt model2.pt model3.pt etc.

See Model Ensembling Tutorial for details:

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Tips for Best Training Results  ☘️ RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Supervisely Ecosystem  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[ifahim]

The models are trained for two separate classes. yolov3 for person and yolov5 for small objects.

[glenn-jocher]

@ifahim oh, if models are trained on seperate classes then there's no automated way to combine their outputs, you can build a custom solution.

[whuyyc]

@glenn-jocher How about using knowledge distillation after ensembling multiple models?

[glenn-jocher]

@whuyyc could you explain a bit more about how your idea would work?

[whuyyc]

@glenn-jocher Although we can ensemble several models to get a better performance than any single model, it will increase computational cost in inference. So can we make a compression for ensembling model using Knowledge Distillation to get a similar performance in a model of smaller size? But I just find the distillation method from one large model(teacher model) to one small model(student model) as shown in this picture: In other words, I do not know how to fuse several models to one model, or directly use KD from ensembling models.... Could you give me some thought or suggestion please?

[akashAD98]

In this kaggle competition they have combined different models using ensamble techniques.
have a look on 1st place solution approach.
https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229724

it will be really grateful if anyone can provide script to combine different object detection models.thanks

[glenn-jocher]

@akashAD98 ensembling is automatically built into YOLOv5. See Model Ensembling tutorial to get started.

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Tips for Best Training Results  ☘️ RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Supervisely Ecosystem  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[glenn-jocher]

@whuyyc I'm not sure the graphic you have there applies, as it shows one trained model and one model undergoing training, whereas the ensembles are normally done with multiple fully trained models. You're free to experiment with different ensembling techniques to explore what might work best. There's even various other ensembling techniques displayed in the Ensemble() module, such as mean ensemble, max ensemble, etc, that can be applied to models with the same output shapes:

yolov5/models/experimental.py

Lines 98 to 111 in c03d590

	class Ensemble(nn.ModuleList):
	# Ensemble of models
	def __init__(self):
	super(Ensemble, self).__init__()
	def forward(self, x, augment=False):
	y = []
	for module in self:
	y.append(module(x, augment)[0])
	# y = torch.stack(y).max(0)[0] # max ensemble
	# y = torch.stack(y).mean(0) # mean ensemble
	y = torch.cat(y, 1) # nms ensemble
	return y, None # inference, train output

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.