Have you ever trained the model without weight decay? #469
Comments
|
@developer0hye ah that's interesting. Recently I have not tested the effect of removing it, but in the past I saw similar results short term with or without it. Darknet applies 0.0005 weight decay, but if they are smart they are only applying it to actual parameters and not the batchnorm2d layers, which brings me to a serious bug we have that is still unresolved: We are applying weight_decay to all parameters, but it should NOT be applied to BatchNorm2d layers. I don't know why PyTorch defaults to this, its an error on their part, leaving us to manually fix somehow. I think the proper way of enforcing this is creating two different parameter groups in the optimizer, so that one group applies to BatchNorm2d layers and the other group to all other parameters, but I have not found an easy way to implement this. There are a few message boards on the topic, such as https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994 |
|
@developer0hye or of course in the absence of a proper implementation perhaps the best alternative is to simply remove it for now. Are your results repeatable, say on one of the smaller datasets like coco_64img.data? |
|
@developer0hye I was thinking I should explain my workflow for testing changes. In the past I used small dataset (i.e. coco_32img.data) results to test and apply changes, but I realized the small datasets were not good indicators of performance on full coco. So then I started using 1 epoch of full COCO as my benchmark, trained at 320 img_size. But then again I realized that epoch 1 improvement does not correlate 100% with full training improvement. So my current workflow is to test ideas on a small repeatable open-source dataset like coco_64img.data 320 img_size for 273 epochs, then if I see improvement there test it on 10% of the full coco training, 27 epochs. The results of some recent experiments using this benchmark are at #441 (comment). So if we wanted to remove weight_decay from the repo (which may be a good idea), the change needs to display improvement on this 27-epoch benchmark. It's a lot of work, but it sets a high bar so only actual improvements enter the system. |
|
Thank you for your great comment. I missed the one point that may be a important information. I used the GroupNorm instead of BatchNorm, so if I used the BatchNorm, the result would get different results. And ... I have one question. It's some theoretical. Why do you think Weight Decay should not be applied to BatchNorm's parameters? |
|
@developer0hye ah, so from a theoretical standpoint, weight decay should only be applied to trained parameters. It's purpose is to prevent overtraining by constantly and slowly dragging the weights to zero (i.e. resetting them to where they started). But the weights and biases in the BatchNorm layers, and probably in the GroupNorm layers also, are not trained, they are simply the running mean of the values that pass through them during each forward pass. So it's impossible to 'overtrain' BatchNorm layers, since they are not trained at all. The result is that weight_decay is harming them every time it is applied, by dragging down the running means towards zero. BTW, I am training on the 27-epoch benchmark with the weight_decay removed. At epoch 18 now, and all metrics (P, R, mAP, F1) look better than default so far, so it looks like this is a good change! I will wait until the training finishes and then update #441 with the result, and commit the change if the final mAP improves. |
|
Ah okay... Hmm if pytorch optimizer decay the bathnorm's parameters for normalization(mean and variance) except parameters for scaling(alpha and beta), it will make a some optimization problem. Thank you for your answer. I hope this issue become the key for higher performance comparable to original darknet YOLO! |
|
@developer0hye the results are in. Default is in blue, removing weight decay is in orange. Orange seems to produce a smoother training, though final mAP does not improve unfortunately, as blue shows much better response during the LR scheduler steps at the end. Also orange produces higher val Objectness loss. I think this is indicative of the overfitting problem we generally have at the moment with objectness. See #453
|
|
Have you guys tried the code below? params_dict = dict(model.named_parameters())
params = []
for key, value in params_dict.items():
print(key)
if 'conv.weight' in key:
params += [{'params':value, 'weight_decay':hyp['weight_decay']}]
else:
params += [{'params':value, 'weight_decay':0.0}] |
|
Then put params in optimizer instead of model.parametrs(), also remember that remove weight_decay in optimizer |
|
@JiefuYuan hmm that's an interesting approach. Yes this might work. Have you tested it? Some of the Conv2D() layers (the ones without BatchNorm layers) also have biases. Do you think the right thing to do is apply the weight_decay to those biases also? |
|
I am training it right now. In my opinion, I don't think we should apply weight_decay to biases since weight decay(l2 regularization) doesn't have any biases in it. What do you think? |
|
@JiefuYuan haha, well it is called weight_decay and not bias_decay, but it's common for people to say 'weights' when they are really talking about 'parameters', so I don't know. I suppose the theory is that they drag parameters back to their starting points actively over time, so if the parameters don't have a good reason for being at their present values they will slowly drift back to zero, where they presumably started from. But at the same time it is fighting against the optimizer updates, especially if the gradient wants to move the parameters away from zero. So I suppose the same theory would hold true for biases as well as weights, since both are initialized about zero by default in pytorch. I'll ask the AlexeyAB this, this is a good question. |
|
@JiefuYuan I found this. So perhaps bias decay is not often used, and then you are right to leave it out. |
|
@JiefuYuan @developer0hye ah of course, this all makes sense now. If I examine the actual BatchNorm2d() values I see the weights are initialized to all 1's, and the biases are all 0's: So we are murdering the BatchNorm2d() weights by applying weight decay to them! Ok this must be our #1 priority presently then to fix this bug. @JiefuYuan I developed an alternative implementation that groups the params into two parameter groups: # Optimizer
pg0, pg1 = [], []
for k, v in dict(model.named_parameters()).items():
if 'Conv2d.weight' in k:
pg1 += [v] # parameter group 1 (apply weight_decay)
else:
pg0 += [v] # parameter group 0
# optimizer = optim.Adam(pg0, lr=hyp['lr0'])
# optimizer = AdaBound(pg0, lr=hyp['lr0'], final_lr=0.1)
optimizer = optim.SGD(pg0, lr=hyp['lr0'], momentum=hyp['momentum'], nesterov=True)
optimizer.add_param_group({'params': pg1, 'weight_decay': hyp['weight_decay']}) # add pg1 with weight_decayWhen I look at the optimizer after this code I see this. I think this is what we want, right? |
I think you are right! |
|
I tested 13 epochs of the update. It seems the large performance difference earlier was due to removing weight decay from the entire model, not simply the bathnorm layers. It appears weight decay on batchnorm layers has little overall effect (blue vs orange). |
|
@glenn-jocher |
|
Have you ever trained the model using Adam optimizer without weight decay? You did these experiments with SGD optimizer, didn't you?? |
|
@developer0hye yes it was one of the first things we tried, but we got consistently worse results. See #4 Note that Adam requires a different LR, but even so we couldn't get it to match SGD performance. |
|
@glenn-jocher |
|
@developer0hye you're welcome! Thanks for all the good PRs! BTW, I just found a new paper that came out with high mAP called xNets, which uses some novel pyramid scheme at different aspect ratios, and it uses Adam at 5E-5 LR to train (!), so I opened up my Adam experiments again, and was able to get better results than SGD on COCO epoch zero. I did a study from 1E-5 to 15E-5 LR, and found the best results at 9E-5 LR. You can see the results here. #4 (comment) It would be great to switch in Adam for SGD, as this would remove one hyperparameter (SGD momentum) from our list of unknowns. |
|
@developer0hye I was able to get improved results with Adam using LR ranges of 5E-5 to 9E-5. This comparison below is at 9E-5 Adam LR. The final mAPs were 44.9 SGD and 45.2 Adam: |
Hi, jocher. Do you remember this issue? |
|
@developer0hye I think the green line is removing weight decay entirely from the model. This results in much faster and better learning during early training (i.e. maybe first 0-100 epochs of COCO training from scratch), but then earlier overfitting and lower best mAP (maybe 1-2% lower). |
|
@developer0hye BTW, if you step back and think about actual real world scenarios, zero weight decay training I think definitely has a place, as it allows you to train a model maybe 2x faster that performs only slightly worse than a typical weight decay model of 0.0005 (from scratch). Fine tuning or transfer learning I'm not sure, haven't explored weight decay effects as much. In transfer learning in general though our YOLO models tend to overfit objectness much faster than the other two losses, and I've never really found a great solution for this (as it's not possible to apply weight decay per loss). Also note that based on the above experiments, we now have 3 parameter groupings for the optimizer, and the batchnorm layers do not have any weight decay at all now. This is here: |
|
@glenn-jocher the orange line is the results removing weight decay for batchnorm layer? I found very interesting post related to this issue. |
|
@developer0hye green is no weight decay at all, with blue applying weight decay to all layers and green to all layers except for batch_norm layers I think. |
You mentioned green two times. I understood final one is the explanation for orange line. Orange reported low performance because the model were no trained sufficiently. Can we expect the orange one finally reachs better performance if we train the model with large iteration??? |
|
Ah sorry, orange and blue are nearly identical because one of them simply has no weight decay on the batchnorm layers. We apply weight decay to all parameters except for batch_norm layer parameters now. This should produce the best results. |
Have you ever trained the model for CoCo dataset without weight decay?
When I trained and tested my model with custom data, using weight decay method harm performance.
Best mAP during 68 epochs
0.95(with weight decay) vs 0.985(w/o weight decay)
The text was updated successfully, but these errors were encountered: