W&B: Proposal for supporting W&B sweeps

[AyushExel]

based on the feature requests, here's a proposal for enabling sweeps.
Steps:

• Set the hyperparameter ranges in data/hyps/sweep.yaml and also define the dataset path and search strategy to be used.
• run wandb sweep utils/wandb_logging/sweep.yaml . You can optionally pass --project name --count max_run_count
• The above command will output an id. Start the sweep with wandb agent sweep_id
  This will start W&B sweep over the defined search space in sweep.yaml. Here's a live example from this script.
  

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Integration of Weights & Biases (W&B) hyperparameter sweeps feature into the YOLOv5 training pipeline.

📊 Key Changes

• 🆕 Added sweep.py, a new script to conduct hyperparameter sweeps via W&B.
• 🛠️ Implemented functionality to parse hyperparameters directly from W&B sweep configuration.
• 📄 Created a W&B sweep configuration file sweep.yaml outlining possible ranges for various hyperparameters.
• 👾 Adjusted wandb_utils.py to utilize the correct batch size during W&B setup.

🎯 Purpose & Impact

• 💹 Provides users the ability to automatically search for the best model hyperparameters, potentially improving model performance.
• ⏱️ Saves time for users by automating the training process with a variety of hyperparameters without manual intervention.
• 📈 Increases model experimentation capabilities, thus potentially fostering more advanced and accurate models.

[glenn-jocher]

@AyushExel thanks for the PR!

I think some of the sweep parameters you hardcoded are suboptimal or will produce Nan trainings, such as 0.99 momentum limit and the higher perspective limits. You should use the same limits and scales (if possible) found in the evolution meta dictionary for best results:

yolov5/train.py

Lines 564 to 594 in dabad57

	# Hyperparameter evolution metadata (mutation scale 0-1, lower_limit, upper_limit)
	meta = {'lr0': (1, 1e-5, 1e-1), # initial learning rate (SGD=1E-2, Adam=1E-3)
	'lrf': (1, 0.01, 1.0), # final OneCycleLR learning rate (lr0 * lrf)
	'momentum': (0.3, 0.6, 0.98), # SGD momentum/Adam beta1
	'weight_decay': (1, 0.0, 0.001), # optimizer weight decay
	'warmup_epochs': (1, 0.0, 5.0), # warmup epochs (fractions ok)
	'warmup_momentum': (1, 0.0, 0.95), # warmup initial momentum
	'warmup_bias_lr': (1, 0.0, 0.2), # warmup initial bias lr
	'box': (1, 0.02, 0.2), # box loss gain
	'cls': (1, 0.2, 4.0), # cls loss gain
	'cls_pw': (1, 0.5, 2.0), # cls BCELoss positive_weight
	'obj': (1, 0.2, 4.0), # obj loss gain (scale with pixels)
	'obj_pw': (1, 0.5, 2.0), # obj BCELoss positive_weight
	'iou_t': (0, 0.1, 0.7), # IoU training threshold
	'anchor_t': (1, 2.0, 8.0), # anchor-multiple threshold
	'anchors': (2, 2.0, 10.0), # anchors per output grid (0 to ignore)
	'fl_gamma': (0, 0.0, 2.0), # focal loss gamma (efficientDet default gamma=1.5)
	'hsv_h': (1, 0.0, 0.1), # image HSV-Hue augmentation (fraction)
	'hsv_s': (1, 0.0, 0.9), # image HSV-Saturation augmentation (fraction)
	'hsv_v': (1, 0.0, 0.9), # image HSV-Value augmentation (fraction)
	'degrees': (1, 0.0, 45.0), # image rotation (+/- deg)
	'translate': (1, 0.0, 0.9), # image translation (+/- fraction)
	'scale': (1, 0.0, 0.9), # image scale (+/- gain)
	'shear': (1, 0.0, 10.0), # image shear (+/- deg)
	'perspective': (0, 0.0, 0.001), # image perspective (+/- fraction), range 0-0.001
	'flipud': (1, 0.0, 1.0), # image flip up-down (probability)
	'fliplr': (0, 0.0, 1.0), # image flip left-right (probability)
	'mosaic': (1, 0.0, 1.0), # image mixup (probability)
	'mixup': (1, 0.0, 1.0), # image mixup (probability)
	'copy_paste': (1, 0.0, 1.0)} # segment copy-paste (probability)

[AyushExel]

@glenn-jocher yeah those hyps are just meant for testing. I'll replace them with same search space from evolution. I just wanted to confirm where you'd like the files sweep.py and sweep.yaml

[AyushExel]

@glenn-jocher I've updated the hyperparameter range. It is now same as evolve search space

[glenn-jocher]

@AyushExel I think this looks good now!

Can you put the sweep.yaml file in utils/wandb_logging to help organize all the W&B assets in one place? Thanks!

[AyushExel]

@glenn-jocher only the sweeps.yaml file or sweep.py file also? I think moving the sweep.py file will create some relative import complications.

[glenn-jocher]

@AyushExel ah I didn't realize sweep.py was in the root directory. Yes we need to move all W&B related files to utils/wandb_logging.

You can see yolo.py for an example of executables in subdirectories. We simply add yolov5/ to the path and everything should work:

yolov5/models/yolo.py

Lines 7 to 23 in b3dabdc

	import argparse
	import logging
	import sys
	from copy import deepcopy
	from pathlib import Path
	FILE = Path(__file__).absolute()
	sys.path.append(FILE.parents[1].as_posix()) # add yolov5/ to path
	from models.common import *
	from models.experimental import *
	from utils.autoanchor import check_anchor_order
	from utils.general import make_divisible, check_file, set_logging
	from utils.plots import feature_visualization
	from utils.torch_utils import time_synchronized, fuse_conv_and_bn, model_info, scale_img, initialize_weights, \
	select_device, copy_attr

[AyushExel]

@glenn-jocher I've changed the location of the files but unfortunately, the issue still persists. This worked fine in the root dir, so I suspect the problem is this -

• the base dir of execution will be utils/wandb_logging/ and in the first line we add the yolov5/ dir to sys path by doing sys.path.append(FILE.parents[2].as_posix())
• Now when in train.py, import test is called, python will first look for any modules named test in the packages and then in the sys directories. It finds the default testing package called test and pops up this error
  

To reproduce:

wandb sweep utils/wandb_logging/sweep.yaml

I'll let you decide how you want to approach this. But I think this might also pop up during pip-package creation. If the user runs training from any other directory, instead of importing the test.py file, python will end up importing the test suite. maybe changing the name of modules to remove the conflicting names might be the right approach. for example changing the name from test.py to yolo_test.py solved this problem. you just have to change test.run() to yolo_test.run()

[glenn-jocher]

@AyushExel can you run it directly from yolov5/?

Are you sure you aren't mixing topics also? If test package import causes a problem then I imagine it would cause a problem for all trainings with this repo, not just the new sweeps functionality.

[AyushExel]

@glenn-jocher this isn't causing problems in training because generally training is done from yolov5/ dir python train.py but sweeps is called like this wandb sweep utils/wandb_logging/sweep.yaml.

In the first case, test.py takes priority over python's test framework in the second case python's test framework takes priority.

[AyushExel]

@glenn-jocher to reproduce this issue:

• Create a new file inside any child folder of the root, for example consider - utils/train_v2.py
• The file should look like this :

import sys
from pathlib import Path

FILE = Path(__file__).absolute()
sys.path.append(FILE.parents[1].as_posix())  # add utils/ to path
import train 

train.run(data="../data/coco128.yaml", hyp="../data/hyps/hyp.scratch.yaml")

• now cd utils
• run python train_v2.py
  This will throw:

  File "/home/jupyter/repos/yolov5/train.py", line 389, in train
    results, maps, _ = test.run(data_dict,
AttributeError: module 'test' has no attribute 'run'

This is because the name test is conflicting when it is not directly in the root yolov5/ folder

This proves that this issue will arise if you try to train from any sub-directories of the root

[glenn-jocher]

@AyushExel right I know, that's why I was asking for you to run it directly from yolov5 rather than from a subfolder, i.e.:

python models/yolo.py rather than cd models && python yolo.py

EDIT: if this works then we can just mention this in the sweep.py Usage example and in a tutorial etc.

[AyushExel]

@glenn-jocher I think I didn't explain the problem properly. Even when running the training from yolov5/ dir, the error occurs. Basically, I'm saying that, the file that calls train should be in the root for import test to work as desired.
python utils/train_v2.py throws the same error.

You can try by pasting this code in utils.train_v2.py :

import sys
from pathlib import Path

FILE = Path(__file__).absolute()
sys.path.append(FILE.parents[1].as_posix())  # add yolov5/ to path
import train 

train.run()

then run python utils/train_v2.py. Does this work for you without errors?

[glenn-jocher]

@AyushExel yes it runs correctly for me:

[AyushExel]

@glenn-jocher That's strange. It throws error only after 1st epoch when it tries to run test.run(). Your epochs executed without this error? I have the same setup here.

[glenn-jocher]

@AyushExel yeah I see what you mean now. This is less of an issue of where to place files than a simple package conflict with test package.

The simplest solution is probably just to rename test.py to val.py to avoid conflicts in a new PR prior to this one.

If you make this change everything works for you right?

[AyushExel]

@glenn-jocher Yup that'd work. I just tested by changing test to val and the same script works :) Should I include that change in this PR?

[glenn-jocher]

@AyushExel no, I'm making the change myself. The naming convention was much more deeply rooted across the repo than I realized, so I'm changing it everywhere. @Borda from pytorch lightning also requested this change last year to avoid conflicts so I think it's a good idea.

See 'Rename test.py to val.py' PR #4000

[glenn-jocher]

@AyushExel good news 😃! Your original issue may now be fixed ✅ in PR #4000. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

[Borda]

@glenn-jocher mind considers refactor the training to PytorchLightnint, then you would get all these logger sand parallel training out of the box...

[AyushExel]

@glenn-jocher Thank you for the quick update. Now this sweep functionality works fine as it is. Running this sweep from this branch. This can be merged now.

[glenn-jocher]

@AyushExel great! Are the tutorial instructions in the first message still valid? #3938 (comment)

[glenn-jocher]

@AyushExel PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

[AyushExel]

@AyushExel great! Are the tutorial instructions in the first message still valid? #3938 (comment)

@glenn-jocher Yes, I've updated the instruction in the first message

[glenn-jocher]

@AyushExel I've updated the Hyperparameter Evolution tutorial with the W&B content in https://docs.ultralytics.com/yolov5/tutorials/hyperparameter_evolution. Let me know if you'd like to update it.

[AyushExel]

@glenn-jocher Thanks. Looks good :)