-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apply FReLU to YOLOv5 #2
Comments
Hi @glenn-jocher , |
@nmaac great, thanks for the feedback! We'll try to experiment with this and see if it helps our results! |
Hi glenn, is there any improment on COCO or VOC dataset about FReLU ? Or how does it perform on the verification set? Thanks ! |
@clw5180 I've run some experiments with YOLOv5s, which seem to show some early improvement. I have not tried larger models, and improvements from one size to another sometimes does not extrapolate well, so I can't draw any conclusions across the entire v5 lineup, but so for small models it appears to help. One warning is that FReLU increases training requirements greatly, similar to Swish perhaps, not quite as bad as Mish, so it's probably not a viable replacement for all activation functions in a model, but might be suitable for partial replacement. |
Thank you for your great contributions! In YOLOv5 we skipped over using Swish or Mish due to their expensive nature, especially during training, and retained the same LeakyReLU(0.1) as YOLOv3. We have a PR for the addition of FReLU
ultralytics/yolov5#556 as this. Is this a correct implementation? Should we use torch.max() or torch.maximum()?
Secondly, I saw you recommended using FReLU only in the backbone of a detection network. So would you recommend replacing all of our LeakyReLU(0.1) activations in the backbone with FReLU() to start, and then doing the same with the head activations if we see positive results? Our main model structure is here:
https://github.com/ultralytics/yolov5/blob/master/models/yolov5s.yaml
Thank you in advance in advance for your time and any recommendations for our situation!
The text was updated successfully, but these errors were encountered: