Corrupt JPEG data

[alicera]

❔Question

If I test the image and it show the message.
How can I debug it?
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9

[glenn-jocher]

@alicera these are hard to track since the error is a low-level C error in some of the functions that cv2 uses. Essentially some of your JPGs are corrupt and not fully saved correctly. The corrupt part will simply be black.

[alicera]

Does it affect the train and test mAP?

[glenn-jocher]

@alicera good news 😃! Your original issue may now be fixed ✅ in PR #3638. This PR adds JPEG corruption error checking by @xiaowk5516 to the YOLOv5 train and testloaders. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[alicera]

Thanks.
How to do the status for code when meet the problem ?

[glenn-jocher]

@alicera the new PR #3638 should identify each corrupted JPG by name.

[abuelgasimsaadeldin]

Hi @glenn-jocher, I am currently facing this issue of "Corrupted JPEG" too and I cannot identify the reason why? Previously I was training on these same exact images and they all appeared as "found" but now all of a sudden most of them are deemed as "corrupted". I have attached one of those images below for your reference. Thanks.

0a7cf41a-6783-4b46-9248-9424b8602924_1.txt

[xiaowk5516]

@abuelgasimsaadeldin hi! jpg and jpeg always end with '0xff','0xd9'. I notice this image ends with '0x00'.

I restore this image using opencv.

now all '0x00' disappear.

[abuelgasimsaadeldin]

Hi @xiaowk5516 hm interesting, so what could be the cause of this issue? The image seems fine and can be read without any problems.

[xiaowk5516]

Hi @xiaowk5516 hm interesting, so what could be the cause of this issue? The image seems fine and can be read without any problems.

I have no idea what makes it. when we parse a picture, we usually end after encountering 0xffd9. parser may ignore the 0x00 following 0xffd9. so it seems fine.

[abuelgasimsaadeldin]

Hi @xiaowk5516 I see, than why is it still considered as "corrupted" if the parser ignores the 0x00 following 0xffd9?

[xiaowk5516]

@abuelgasimsaadeldin hi, do all your 'corrupted' images exist some data following 0xffd9? This is not a common situation.

[glenn-jocher]

@xiaowk5516 @abuelgasimsaadeldin I tested #3638 on a variety of our autodownload datasets (VOC, COCO, SKU-110k, VisDrone, etc), and almost none of them showed any corrupted JPEGs other than SKU, which already reported corrupt images before the PR.

COCO notably has zero out of 120k corrupted, so if there is a warning about your images it's likely actually a problem in your images. If they appear ok visually you might simply load and save them again with cv2.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[glenn-jocher]

@alicera good news 😃! Your original issue may now be fixed ✅ in PR #4548. This PR automatically restores and saves corrupted JPEGs before training starts, and all images are now used for training, including the restored JPEGs.

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!