Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training stop when load_image throw error #165

Closed
Matanelc opened this issue Jun 22, 2020 · 5 comments
Closed

Training stop when load_image throw error #165

Matanelc opened this issue Jun 22, 2020 · 5 comments
Labels
bug Something isn't working

Comments

@Matanelc
Copy link

Step to reproduce
1.start train
2.have "0" size image.
or
1.start train
2. have missing label for images
in those cases the train is stoped with error :
data = [self.dataset[idx] for idx in possibly_batched_index] File "/app/workspace/utils/datasets.py", line 451, in __getitem__ img, labels = load_mosaic(self, index) File "/app/workspace/utils/datasets.py", line 574, in load_mosaic img, _, (h, w) = load_image(self, index) File "/app/workspace/utils/datasets.py", line 536, in load_image assert img is not None, 'Image Not Found ' + path AssertionError: Image Not Found /app/workspace/data/BBFashion/images/5e4118b37ddc9d75987fa7f3.jpg
why the image is not just getting "skipped" ?

@Matanelc Matanelc added the bug Something isn't working label Jun 22, 2020
@github-actions
Copy link
Contributor

github-actions bot commented Jun 22, 2020

Hello @Matanelc, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook Open In Colab, Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

  • Cloud-based AI systems operating on hundreds of HD video streams in realtime.
  • Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
  • Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

@Al-Razi-KR
Copy link

I also have the same error, even though none of my images are nonetype nor my labels are empty.

@glenn-jocher
Copy link
Member

@Matanelc I'm not sure why I have to explain this to you, but 0 size images will obviously not work correctly. There is no bug.

@da2r-20
Copy link

da2r-20 commented Jun 25, 2020

@Matanelc I'm not sure why I have to explain this to you, but 0 size images will obviously not work correctly. There is no bug.

I know that in darknet the training script just ignores bad annotations and bad images.
The bottom line when dealing with large data there is always a chance of several images going bad.

Thus a training script should be robust and handle these errors instead of making the training fail after a long time of running.

It was also a problem here:
#195

and about the response above
This is open source, people will open bugs because this is how a community works.
You should either grow up and embrace it or take the repo down.

@glenn-jocher
Copy link
Member

glenn-jocher commented Jun 25, 2020

@doronAtuar facilitating silent errors is never best practices.

https://zen-of-python.info/errors-should-never-pass-silently.html
Errors should never pass silently. Unless explicitly silenced.
Just because programmers often ignore error messages doesn’t mean the program should stop emitting them. Silent errors can happen when functions return error codes or None instead of raising exceptions. These two aphorisms tell us that it’s better for a program to fail fast and crash than to silence the error and continue running the program. The bugs that inevitably happen later on will be harder to debug since they are far removed from the original cause. Though you can always choose to explicitly ignore the errors your programs cause, just be sure you are making the conscious choice to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants