master branch warmup learning rate is not bias lr falls from 0.1 to lr0

[challow0]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

No response

Bug

• when adding to optimizer groups, g[2] is not the last to add. But in warmup it is still j == 2

• This is the master branch
  

  yolov5/train.py

  Lines 153 to 171 in 0537e8d

  	g = [], [], [] # optimizer parameter groups
  	bn = tuple(v for k, v in nn.__dict__.items() if 'Norm' in k) # normalization layers, i.e. BatchNorm2d()
  	for v in model.modules():
  	if hasattr(v, 'bias') and isinstance(v.bias, nn.Parameter): # bias
  	g[2].append(v.bias)
  	if isinstance(v, bn): # weight (no decay)
  	g[1].append(v.weight)
  	elif hasattr(v, 'weight') and isinstance(v.weight, nn.Parameter): # weight (with decay)
  	g[0].append(v.weight)
  	if opt.optimizer == 'Adam':
  	optimizer = Adam(g[2], lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	elif opt.optimizer == 'AdamW':
  	optimizer = AdamW(g[2], lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	else:
  	optimizer = SGD(g[2], lr=hyp['lr0'], momentum=hyp['momentum'], nesterov=True)
  	optimizer.add_param_group({'params': g[0], 'weight_decay': hyp['weight_decay']}) # add g0 with weight_decay
  	optimizer.add_param_group({'params': g[1]}) # add g1 (BatchNorm2d weights)

• This is the v6.1 tag branch
  

  yolov5/train.py

  Lines 156 to 173 in 3752807

  	g0, g1, g2 = [], [], [] # optimizer parameter groups
  	for v in model.modules():
  	if hasattr(v, 'bias') and isinstance(v.bias, nn.Parameter): # bias
  	g2.append(v.bias)
  	if isinstance(v, nn.BatchNorm2d): # weight (no decay)
  	g0.append(v.weight)
  	elif hasattr(v, 'weight') and isinstance(v.weight, nn.Parameter): # weight (with decay)
  	g1.append(v.weight)
  	if opt.optimizer == 'Adam':
  	optimizer = Adam(g0, lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	elif opt.optimizer == 'AdamW':
  	optimizer = AdamW(g0, lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	else:
  	optimizer = SGD(g0, lr=hyp['lr0'], momentum=hyp['momentum'], nesterov=True)
  	optimizer.add_param_group({'params': g1, 'weight_decay': hyp['weight_decay']}) # add g1 with weight_decay
  	optimizer.add_param_group({'params': g2}) # add g2 (biases)

• This is the master branch warmup
  

  yolov5/train.py

  Lines 336 to 338 in 0537e8d

  	for j, x in enumerate(optimizer.param_groups):
  	# bias lr falls from 0.1 to lr0, all other lrs rise from 0.0 to lr0
  	x['lr'] = np.interp(ni, xi, [hyp['warmup_bias_lr'] if j == 2 else 0.0, x['initial_lr'] * lf(epoch)])

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@challow0 thanks for the bug report! It looks like you are correct, 338 should have been updated and has not been.

[glenn-jocher]

@challow0 good news 😃! Your original issue may now be fixed ✅ in PR #8356. This sets the bias LR warmup to index 0 which aligns with the bias param group.

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!