Add support for W&B bounding box debugging and metric logging #1108
Conversation
Add Support for W&B Bounding Box debugging and Metric logging
![github-actions[bot]](https://github.com/avatars/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hello @AyushExel, thank you for submitting a PR! To allow your work to be integrated as seamlessly as possible, we advise you to:
- Verify your PR is up-to-date with origin/master. If your PR is behind origin/master update by running the following, replacing 'feature' with the name of your local branch:
git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
git checkout feature # <----- replace 'feature' with local branch name
git rebase upstream/master
git push -u origin -f- Verify all Continuous Integration (CI) checks are passing.
- Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee
|
@AyushExel very interesting, thanks for the PR! I was not aware of this logging tool. One thing I noticed is that you have a ceiling in place on box quantity. An alternative approach which would allow you to remove this argument, which we currently use for inference and for plotting predictions (i.e. test_batch0.jpg) is to only plot boxes above a sensible real-world confidence: Lines 1078 to 1081 in 5fac5ad Not to be confused with the mAP confidence threshold, which is set as close to zero as feasible to record the highest mAP: Line 257 in 5fac5ad Also, a bit on a separate topic but based on your training plots you have may not have trained sufficient epochs. One clue to this (other than validation confidences that do not overfit) is low confidences on your detections (i.e. around 40% confidence in your road traffic gif). |
|
@glenn-jocher I ceiling is for the number of images being logged per epoch and not on the number of bounding boxes. All the detected bounding boxes are being logged. Which means that at maximum 100 images will be logged per epoch. This serves 2 purposes:
Also, plotting all the boxes also serves a purpose. The bounding box debugger of W&B supports the tuning of confidence scores in real-time( Like the GIF above). So, I thought applying a filter would not make sense. But we can relax this restriction in the future. Would love your thoughts on this as well. As for the training, I trained all the models for 100 epochs to keep the comparison fair. But I'm training again for 200 epochs to improve confidence. |
|
@AyushExel ah, got it, thanks for the clarification. We've found image logging ops to be very costly operations, for wall clock time as well as hard drive space, hence the current approach of only plotting the first few batches of the first and final epoch during training. The slowdown is more pronounced for smaller datasets, like the COCO128 tutorial, where each epoch may only last a few seconds, but will affect all datasets to varying degrees naturally. Toeing this line is part of the compromise we are trying to keep, between adding features and speeding up training. |
|
@glenn-jocher Here are some results of the profiling tests that I ran for wandb logging code Time unit => 1e-3 seconds(ms)Time profile for all steps in W&B image logging TIme Profile for all steps in existing image plotting Summary:
One thing to note here is that this time taken will remain constant on a given system, as we're logging a constant number of images( 50 in this case) . In the case of smaller datasets( like coco-128), the penalty will be noticeable but intuitively, we should never log around 50% of the dataset for debugging. But for larger datasets, this penalty won't be that noticeable. But in either case, the number of images to be logged should be available as an argparse parameter to provide full control to the user |
|
@AyushExel thanks for the profile info! Before going any further we'll want to rebase this against master, as there have been many commits since the PR was originally submitted. We have an action that can do this automatically sometimes, I'll try it, but failing that you'll likely need to do this by hand by following the directions in #1108 (review). /rebase |
|
@AyushExel ok, I ran some speed tests for an alternative implementation. If you put a breakpoint at L146 of test.py (after your rebase) and test the following lines you'll see that the second accomplishes the same as the first in about 2% of the time (94us vs 5540us, or 50X faster). So you should switch to If wandb can accept a python %timeit [[float(x1), float(x2), float(x3), float(x4), float(x5), float(x6)] for x1, x2, x3, x4, x5, x6 in pred.clone()]
5.54 ms ± 33.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit [[x1, x2, x3, x4, x5, x6] for x1, x2, x3, x4, x5, x6 in pred.clone().tolist()]
94.1 µs ± 409 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) |
|
@glenn-jocher Thanks for pointing this out. I was not able to merge this branch with the updated version of the master branch as there were a lot of conflicts. I ended up with broken code when trying to resolve the conflicts. So, I started fresh with a clean branch and made a new PR with the changes you requested. Here the new PR #1235 . Feel to close this PR. |
That's a good idea. I was actually thinking of recommending that. Will close. |
This PR adds support for debugging models using W&B, only if the library is already found installed. When using W&B, users of YOLOV5 can debug their models easily inside a customizable dashboard by logging & comparing performance metrics, system usage metrics (like GPU memory) and predictions.
W&B is free for all individuals, academics and open source projects
Here is a live dashboard comparing all the standard YOLOv5 models
Features:
Bonding Box Debugging
Debug you bounding box predictions in real-time.
Automatically log and compare the performance of multiple models
Supports Resuming
When training is resumed from a previous checkpoint, the metrics and images will continue to be logged in the same W&B dashboard if it exists, otherwise, a new W&B run will be created
Adds no dependencies
The library will work as it is supposed to if
wandbis not installed and will only log metrics and media files to W&B if it is installed. To enable W&B logging, you just need to install the library usingpip install wandbTo disable logging you can set
os.environ['WANDB_DISABLED'] = 'true'in your code or callwandb offusing the command line.🛠️ PR Summary
Made with ❤️ by Ultralytics Actions
🌟 Summary
Enhanced YOLOv5 testing and training modules with improved logging, code refactoring, and Weights & Biases integration.
📊 Key Changes
save_confandnum_predictionsarguments to control output during testing.🎯 Purpose & Impact
Users will experience a more seamless testing and training process with enhanced tools for monitoring their model's performance, leading to a smoother development cycle and potential improvements in model accuracy.