How to increase performance in CBIS-DDSM?

[flashcp]

❔Question

hello, @glenn-jocher
I would like to apply yolov5 to cbis-ddsm, but can`t get good results. How should I modify the hyper-parameter or make some adjustments on datasets?
thanks.

Additional context

datasets

The CBIS-DDSM (Curated Breast Imaging Subset of DDSM) is an updated and standardized version of the Digital Database for Screening Mammography (DDSM).
In my experiments, I just detected the masses , so there was only one category. the train datasets contains 1231 images and test datasets contains 361 images and set up the datasets structure as required by "Train Custom Data Tutorial"
TensorFlow already includes CBIS-DDSM. There are description, api and download link about CBIS-DDSM.

Modification

Because the datasets is small, I modify the warmup iterations from 1000 to 500.
nw = max(round(hyp['warmup_epochs'] * nb), 500) # number of warmup iterations, max(3 epochs, 1k iterations)
In addition, the medical image unlike nature image, it not have hsv just a gray scale image. So I disable image HSV augmentation in LoadImagesAndLabels. Does this cause any errors or affect performance?

hyp and results

config:
--weights '' --hyp data/hyp.scratch.yaml --cfg yolov5s.yaml --data data/cbis_ddsm.yaml --batch-size 12 --epochs 100 --device 0 --img-size 1024

hyp.yaml:
lr0: 0.01 lrf: 0.2 momentum: 0.937 weight_decay: 0.0005 warmup_epochs: 3.0 warmup_momentum: 0.8 warmup_bias_lr: 0.1 box: 0.05 cls: 0.5 cls_pw: 1.0 obj: 1.0 obj_pw: 1.0 iou_t: 0.2 anchor_t: 4.0 fl_gamma: 0.0 hsv_h: 0.015 hsv_s: 0.7 hsv_v: 0.4 degrees: 0.0 translate: 0.1 scale: 0.5 shear: 0.0 perspective: 0.0 flipud: 0.0 fliplr: 0.5 mosaic: 1.0 mixup: 0.0

Thanks again for your work and look forward to you reply.

[github-actions]

👋 Hello @flashcp, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@flashcp please consider submitting a PR to add the CBIS-DDSM dataset to our autodownload-capable list of datasets to help us reproduce your issue and to help future users get started more easily with this dataset. Example PR is #2968. Glancing at your results your model is too small, you are not starting from pretrained weights, you are not training enough epochs, and you may want to evolve your hyperparameters. A full guide on improving training results is below.

• COCO
• COCO128
• VOC
• Argoverse-HD
• VisDrone2019-DET
• GlobalWheat-2020
• KITTI
• SKU110 (grocery store items)
• CityScapes
• Objects365
• OpenImages

👋 Hello! Thanks for asking about improving training results. Most of the time good results can be obtained with no changes to the models or training settings, provided your dataset is sufficiently large and well labelled. If at first you don't get good results, there are steps you might be able to take to improve, but we always recommend users first train with all default settings before considering any changes. This helps establish a performance baseline and spot areas for improvement.

If you have questions about your training results we recommend you provide the maximum amount of information possible if you expect a helpful response, including results plots (train losses, val losses, P, R, mAP), PR curve, confusion matrix, training mosaics, test results and dataset statistics images such as labels.png. All of these are located in your project/name directory, typically yolov5/runs/train/exp.

We've put together a full guide for users looking to get the best results on their YOLOv5 trainings below.

Dataset

• Images per class. ≥1.5k images per class
• Instances per class. ≥10k instances (labeled objects) per class total
• Image variety. Must be representative of deployed environment. For real-world use cases we recommend images from different times of day, different seasons, different weather, different lighting, different angles, different sources (scraped online, collected locally, different cameras) etc.
• Label consistency. All instances of all classes in all images must be labelled. Partial labelling will not work.
• Label accuracy. Labels must closely enclose each object. No space should exist between an object and it's bounding box. No objects should be missing a label.
• Background images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total).

Model Selection

Larger models like YOLOv5x and YOLOv5x6 will produce better results in nearly all cases, but have more parameters, require more CUDA memory to train, and are slower to run. For mobile deployments we recommend YOLOv5s/m, for cloud deployments we recommend YOLOv5l/x. See our README table for a full comparison of all models.

• Start from Pretrained weights. Recommended for small to medium sized datasets (i.e. VOC, VisDrone, GlobalWheat). Pass the name of the model to the --weights argument. Models download automatically from the latest YOLOv5 release.

python train.py --data custom.yaml --weights yolov5s.pt
                                             yolov5m.pt
                                             yolov5l.pt
                                             yolov5x.pt

• Start from Scratch. Recommended for large datasets (i.e. COCO, Objects365, OIv6). Pass the model architecture yaml you are interested in, along with an empty --weights '' argument:

python train.py --data custom.yaml --weights '' --cfg yolov5s.yaml
                                                      yolov5m.yaml
                                                      yolov5l.yaml
                                                      yolov5x.yaml

Training Settings

Before modifying anything, first train with default settings to establish a performance baseline. A full list of train.py settings can be found in the train.py argparser.

• Epochs. Start with 300 epochs. If this overfits early then you can reduce epochs. If overfitting does not occur after 300 epochs, train longer, i.e. 600, 1200 etc epochs.
• Image size. COCO trains at native resolution of --img 640, though due to the high amount of small objects in the dataset it can benefit from training at higher resolutions such as --img 1280. If there are many small objects then custom datasets will benefit from training at native or higher resolution. Best inference results are obtained at the same --img as the training was run at, i.e. if you train at --img 1280 you should also test and detect at --img 1280.
• Batch size. Use the largest --batch-size that your hardware allows for. Small batch sizes produce poor batchnorm statistics and should be avoided.
• Hyperparameters. Default hyperparameters are in hyp.scratch.yaml. We recommend you train with default hyperparameters first before thinking of modifying any. In general, increasing augmentation hyperparameters will reduce and delay overfitting, allowing for longer trainings and higher final mAP. Reduction in loss component gain hyperparameters like hyp['obj'] will help reduce overfitting in those specific loss components. For an automated method of optimizing these hyperparameters, see our Hyperparameter Evolution Tutorial.

Further Reading

If you'd like to know more a good place to start is Karpathy's 'Recipe for Training Neural Networks', which has great ideas for training that apply broadly across all ML domains:
http://karpathy.github.io/2019/04/25/recipe/

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[Chenzkun]

hello,@glenn-jocher，@flashcpProblems like upstairs, I use the model is yolov5m6, excuse me ，how to solve

[Chenzkun]

thanks！

[glenn-jocher]

@Chenzkun I'm sorry I don't understand your question, can you be more specific?

[Chenzkun]

@Chenzkun对不起，我不明白你的问题，你能更具体一点吗？

I used the YOLOV5M6 to run the cbis-ddsm breast cancer data set, but after running 300,500 epochs, the results were not very good. The results were similar to the picture below. What should I do?
Thank you

[glenn-jocher]

@Chenzkun 👋 Hello! Thanks for asking about improving YOLOv5 🚀 training results. The CBIS-DDSM dataset may simply be very hard to train on, as personally I don't think I could spot the objects labelled myself. Nevertheless we have some best practices guidelines below.

Most of the time good results can be obtained with no changes to the models or training settings, provided your dataset is sufficiently large and well labelled. If at first you don't get good results, there are steps you might be able to take to improve, but we always recommend users first train with all default settings before considering any changes. This helps establish a performance baseline and spot areas for improvement.

If you have questions about your training results we recommend you provide the maximum amount of information possible if you expect a helpful response, including results plots (train losses, val losses, P, R, mAP), PR curve, confusion matrix, training mosaics, test results and dataset statistics images such as labels.png. All of these are located in your project/name directory, typically yolov5/runs/train/exp.

We've put together a full guide for users looking to get the best results on their YOLOv5 trainings below.

Dataset

• Images per class. ≥ 1500 images per class recommended
• Instances per class. ≥ 10000 instances (labeled objects) per class recommended
• Image variety. Must be representative of deployed environment. For real-world use cases we recommend images from different times of day, different seasons, different weather, different lighting, different angles, different sources (scraped online, collected locally, different cameras) etc.
• Label consistency. All instances of all classes in all images must be labelled. Partial labelling will not work.
• Label accuracy. Labels must closely enclose each object. No space should exist between an object and it's bounding box. No objects should be missing a label.
• Label verification. View train_batch*.jpg on train start to verify your labels appear correct, i.e. see example mosaic.
• Background images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.

Model Selection

Larger models like YOLOv5x and YOLOv5x6 will produce better results in nearly all cases, but have more parameters, require more CUDA memory to train, and are slower to run. For mobile deployments we recommend YOLOv5s/m, for cloud deployments we recommend YOLOv5l/x. See our README table for a full comparison of all models.

• Start from Pretrained weights. Recommended for small to medium sized datasets (i.e. VOC, VisDrone, GlobalWheat). Pass the name of the model to the --weights argument. Models download automatically from the latest YOLOv5 release.

python train.py --data custom.yaml --weights yolov5s.pt
                                             yolov5m.pt
                                             yolov5l.pt
                                             yolov5x.pt
                                             custom_pretrained.pt

• Start from Scratch. Recommended for large datasets (i.e. COCO, Objects365, OIv6). Pass the model architecture yaml you are interested in, along with an empty --weights '' argument:

python train.py --data custom.yaml --weights '' --cfg yolov5s.yaml
                                                      yolov5m.yaml
                                                      yolov5l.yaml
                                                      yolov5x.yaml

Training Settings

Before modifying anything, first train with default settings to establish a performance baseline. A full list of train.py settings can be found in the train.py argparser.

• Epochs. Start with 300 epochs. If this overfits early then you can reduce epochs. If overfitting does not occur after 300 epochs, train longer, i.e. 600, 1200 etc epochs.
• Image size. COCO trains at native resolution of --img 640, though due to the high amount of small objects in the dataset it can benefit from training at higher resolutions such as --img 1280. If there are many small objects then custom datasets will benefit from training at native or higher resolution. Best inference results are obtained at the same --img as the training was run at, i.e. if you train at --img 1280 you should also test and detect at --img 1280.
• Batch size. Use the largest --batch-size that your hardware allows for. Small batch sizes produce poor batchnorm statistics and should be avoided.
• Hyperparameters. Default hyperparameters are in hyp.scratch-low.yaml. We recommend you train with default hyperparameters first before thinking of modifying any. In general, increasing augmentation hyperparameters will reduce and delay overfitting, allowing for longer trainings and higher final mAP. Reduction in loss component gain hyperparameters like hyp['obj'] will help reduce overfitting in those specific loss components. For an automated method of optimizing these hyperparameters, see our Hyperparameter Evolution Tutorial.

Further Reading

If you'd like to know more a good place to start is Karpathy's 'Recipe for Training Neural Networks', which has great ideas for training that apply broadly across all ML domains: http://karpathy.github.io/2019/04/25/recipe/

Good luck 🍀 and let us know if you have any other questions!

[Chenzkun]

@Chenzkun 👋你好！感谢您询问有关改进 YOLOv5 的问题🚀训练结果。CBIS-DDSM 数据集可能很难训练，因为我个人认为我无法发现标记为自己的对象。不过，我们在下面有一些最佳实践指南。

大多数情况下，只要您的数据集足够大且标记良好，就可以在不更改模型或训练设置的情况下获得良好的结果。如果一开始您没有得到好的结果，您可以采取一些步骤来改进，但我们始终建议用户在考虑任何更改之前**先使用所有默认设置进行训练。**这有助于建立性能基准并找出需要改进的地方。

如果您对训练结果有疑问，**我们建议您尽可能提供尽可能多的信息，**如果您希望得到有用的回复，包括结果图（训练损失、val 损失、P、R、mAP）、PR 曲线、混淆矩阵、训练马赛克，测试结果和数据集统计图像，如labels.png。所有这些都位于您的project/name目录中，通常是yolov5/runs/train/exp.

我们为希望在下面的 YOLOv5 培训中获得最佳结果的用户整理了一份完整指南。

数据集

• **每个班级的图像。**推荐每班≥1500张图像
• **每个类的实例。**推荐每类≥ 10000 个实例（标记对象）
• **图像多样性。**必须代表已部署的环境。对于现实世界的用例，我们推荐来自一天中不同时间、不同季节、不同天气、不同照明、不同角度、不同来源（在线抓取、本地收集、不同相机）等的图像。
• **标签一致性。**必须标记所有图像中所有类的所有实例。部分标签将不起作用。
• **标签准确性。**标签必须紧紧地包围每个对象。对象与其边界框之间不应存在空间。任何对象都不应缺少标签。
• **标签验证。**查看train_batch*.jpg火车开始以验证您的标签是否正确，即查看示例马赛克。
• **背景图像。**背景图像是没有添加到数据集以减少误报 (FP) 的对象的图像。我们建议使用大约 0-10% 的背景图片来帮助减少 FP（COCO 有 1000 张背景图片供参考，占总数的 1%）。背景图像不需要标签。

型号选择

YOLOv5x 和YOLOv5x6等较大的模型几乎在所有情况下都会产生更好的结果，但参数更多，需要更多的 CUDA 内存来训练，并且运行速度更慢。对于移动部署我们推荐 YOLOv5s/m，对于云部署我们推荐 YOLOv5l/x。有关所有模型的完整比较，请参阅我们的 README表。

• **从预训练的权重开始。**推荐用于中小型数据集（即VOC、VisDrone、GlobalWheat）。将模型的名称传递给--weights参数。模型从最新的 YOLOv5 版本自动下载。

python train.py --data custom.yaml --weights yolov5s.pt
                                             yolov5m.pt
                                             yolov5l.pt
                                             yolov5x.pt
                                             custom_pretrained.pt

• **白手起家。**推荐用于大型数据集（即COCO、Objects365、OIv6）。传递您感兴趣的模型架构 yaml，以及一个空--weights ''参数：

python train.py --data custom.yaml --weights '' --cfg yolov5s.yaml
                                                      yolov5m.yaml
                                                      yolov5l.yaml
                                                      yolov5x.yaml

训练设置

在修改任何内容之前，首先使用默认设置进行训练以建立性能基线。train.py 设置的完整列表可以在train.py argparser 中找到。

• **时代。**从 300 个 epoch 开始。如果这过早地过拟合，那么您可以减少时期。如果在 300 个 epoch 之后没有出现过拟合，则训练更长的时间，即 600、1200 等 epoch。
• **图片尺寸。**COCO 以 0 的原始分辨率进行训练--img 640，尽管由于数据集中的大量小对象，它可以从更高分辨率的训练中受益，例如--img 1280. 如果有很多小对象，那么自定义数据集将受益于本机或更高分辨率的训练。最好的推理结果是在--img训练运行的同时获得的，即如果你在训练时--img 1280也应该在 时进行测试和检测--img 1280。
• **批量大小。**使用--batch-size您的硬件允许的最大容量。小批量会产生较差的批量标准统计数据，应避免使用。
• **超参数。**默认超参数在hyp.scratch-low.yaml中。我们建议您在考虑修改任何超参数之前先使用默认超参数进行训练。一般来说，增加增强超参数将减少和延迟过度拟合，从而允许更长的训练时间和更高的最终 mAP。损失分量增益超参数的hyp['obj']减少将有助于减少那些特定损失分量的过度拟合。有关优化这些超参数的自动化方法，请参阅我们的超参数演进教程。

延伸阅读

如果您想了解更多信息，不妨从 Karpathy 的“训练神经网络的食谱”开始，它提供了广泛适用于所有 ML 领域的训练的好主意：http: //karpathy.github.io/2019/04 /25/食谱/

祝你好运🍀如果您还有其他问题，请告诉我们！

I used cbis-ddsm to check that each class has less than 1,500 images and fewer than 10,000 labels. How Do I meet the Yolov5 dataset requirements, thank you

[glenn-jocher]

@Chenzkun oh the above are not hard requirements, they are best practices that you should attempt to reach to produce similar results to the COCO model results etc.

[Chenzkun]

hello，@glenn-jocher
The model I am using is yolov5m6 with 16g of video memory and a batch-size of 2. When I apply the mosaic enhancement, train_batch0.jpg shows the image shown below.

Why is the other half blank?

[Chenzkun]

你好，@glenn-jocher 我使用的模型是 yolov5m6，显存为 16g，batch-size 为 2。当我应用马赛克增强时，train_batch0.jpg 显示如下图所示。 为什么另一半是空白的？

[glenn-jocher]

@Chenzkun batch size 2 is much smaller than recommended, this might be a reason for underperformance. I don't know what your question is regarding mosaics, these are simply 4 images mosaiced together with augmentations.