Enable direct `--weights URL` definition #3373

glenn-jocher · 2021-05-28T11:30:33Z

@kalenmike this PR will enable direct --weights URL definition. Example usage:

python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt
python test.py --weights https://storage.googleapis.com/bucket/dir/model.pt
python detect.py --weights https://storage.googleapis.com/bucket/dir/model.pt

# Hub
model = torch.hub.load('ultralytics/yolov5', 'custom', 'https://storage.googleapis.com/bucket/dir/model.pt')

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Refinement of model downloading process in Ultralytics YOLOv5.

📊 Key Changes

Simplified the code to download pre-trained models by merging the download and loading steps.
Introduced safe_download function for more robust file downloading with error handling and retries.
Ensured the attempt_download function supports direct HTTP/HTTPS URL downloads.

🎯 Purpose & Impact

Improved Reliability: Incorporated more reliable file download checks, reducing the instances of incomplete downloads.
Simplified Error Handling: New function streamlines the downloading process, which catches and handles errors more efficiently.
Easier Debugging: Descriptive error messages will make it easier for both developers and users to troubleshoot download issues.
Direct URL Support: Enhancements include the ability to download files directly from specified URLs, facilitating easier access to pre-trained models.
User Experience: These changes can lead to a smoother and more dependable setup process for users working with YOLOv5 models, minimizing disruptions due to download errors.

@kalenmike

@kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ```

glenn-jocher · 2021-05-28T11:33:10Z

When combined with #3330 we can now pass URLs directly to our four principal arguments that require files:

python train.py --data URL1 --cfg URL2 --hyp URL3 --weights URL4

glenn-jocher · 2021-05-28T13:19:24Z

@kalenmike @SkalskiP PR is merged. This should allow for --weights from URL/remote models now. Usage examples:

python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt
python test.py --weights https://storage.googleapis.com/bucket/dir/model.pt
python detect.py --weights https://storage.googleapis.com/bucket/dir/model.pt

# Hub
model = torch.hub.load('ultralytics/yolov5', 'custom', 'https://storage.googleapis.com/bucket/dir/model.pt')

@kalenmike

* update ci-testing.yml (#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * update ci-testing.yml (#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * Enable direct `--weights URL` definition (#3373) * Enable direct `--weights URL` definition @kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ``` * cleanup * bug fixes * weights = attempt_download(weights) * Update experimental.py * Update hubconf.py * return bug fix * comment mirror * min_bytes * Update tutorial.ipynb (#3368) add Open in Kaggle badge * `cv2.imread(img, -1)` for IMREAD_UNCHANGED (#3379) * Update datasets.py * comment Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * COCO evolution fix (#3388) * COCO evolution fix * cleanup * update print * print fix * Create `is_pip()` function (#3391) Returns `True` if file is part of pip package. Useful for contextual behavior modification. ```python def is_pip(): # Is file in a pip package? return 'site-packages' in Path(__file__).absolute().parts ``` * Revert "`cv2.imread(img, -1)` for IMREAD_UNCHANGED (#3379)" (#3395) This reverts commit 21a9607. * Update FLOPs description (#3422) * Update README.md * Changing FLOPS to FLOPs. Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> * Parse URL authentication (#3424) * Parse URL authentication * urllib.parse.unquote() * improved error handling * improved error handling * remove %3F * update check_file() * Add FLOPs title to table (#3453) * Suppress jit trace warning + graph once (#3454) * Suppress jit trace warning + graph once Suppress harmless jit trace warning on TensorBoard add_graph call. Also fix multiple add_graph() calls bug, now only on batch 0. * Update train.py * Update MixUp augmentation `alpha=beta=32.0` (#3455) Per VOC empirical results #3380 (comment) by @developer0hye * Add `timeout()` class (#3460) * Add `timeout()` class * rearrange order * Faster HSV augmentation (#3462) remove datatype conversion process that can be skipped * Add `check_git_status()` 5 second timeout (#3464) * Add check_git_status() 5 second timeout This should prevent the SSH Git bug that we were discussing @kalenmike * cleanup * replace timeout with check_output built-in timeout * Improved `check_requirements()` offline-handling (#3466) Improve robustness of `check_requirements()` function to offline environments (do not attempt pip installs when offline). * Add `output_names` argument for ONNX export with dynamic axes (#3456) * Add output names & dynamic axes for onnx export Add output_names and dynamic_axes names for all outputs in torch.onnx.export. The first four outputs of the model will have names output0, output1, output2, output3 * use first output only + cleanup Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Revert FP16 `test.py` and `detect.py` inference to FP32 default (#3423) * fixed inference bug ,while use half precision * replace --use-half with --half * replace space and PEP8 in detect.py * PEP8 detect.py * update --half help comment * Update test.py * revert space Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Add additional links/resources to stale.yml message (#3467) * Update stale.yml * cleanup * Update stale.yml * reformat * Update stale.yml HUB URL (#3468) * Stale `github.actor` bug fix (#3483) * Explicit `model.eval()` call `if opt.train=False` (#3475) * call model.eval() when opt.train is False call model.eval() when opt.train is False * single-line if statement * cleanup Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * check_requirements() exclude `opencv-python` (#3495) Fix for 3rd party or contrib versions of installed OpenCV as in #3494. * Earlier `assert` for cpu and half option (#3508) * early assert for cpu and half option early assert for cpu and half option * Modified comment Modified comment * Update tutorial.ipynb (#3510) * Reduce test.py results spacing (#3511) * Update README.md (#3512) * Update README.md Minor modifications * 850 width * Update greetings.yml revert greeting change as PRs will now merge to master. Co-authored-by: Piotr Skalski <SkalskiP@users.noreply.github.com> Co-authored-by: SkalskiP <piotr.skalski92@gmail.com> Co-authored-by: Peretz Cohen <pizzaz93@users.noreply.github.com> Co-authored-by: tudoulei <34886368+tudoulei@users.noreply.github.com> Co-authored-by: chocosaj <chocosaj@users.noreply.github.com> Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> Co-authored-by: Yonghye Kwon <developer.0hye@gmail.com> Co-authored-by: Sam_S <SamSamhuns@users.noreply.github.com> Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: edificewang <609552430@qq.com>

@kalenmike

* update ci-testing.yml (ultralytics#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * update ci-testing.yml (ultralytics#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * Enable direct `--weights URL` definition (ultralytics#3373) * Enable direct `--weights URL` definition @kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ``` * cleanup * bug fixes * weights = attempt_download(weights) * Update experimental.py * Update hubconf.py * return bug fix * comment mirror * min_bytes * Update tutorial.ipynb (ultralytics#3368) add Open in Kaggle badge * `cv2.imread(img, -1)` for IMREAD_UNCHANGED (ultralytics#3379) * Update datasets.py * comment Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * COCO evolution fix (ultralytics#3388) * COCO evolution fix * cleanup * update print * print fix * Create `is_pip()` function (ultralytics#3391) Returns `True` if file is part of pip package. Useful for contextual behavior modification. ```python def is_pip(): # Is file in a pip package? return 'site-packages' in Path(__file__).absolute().parts ``` * Revert "`cv2.imread(img, -1)` for IMREAD_UNCHANGED (ultralytics#3379)" (ultralytics#3395) This reverts commit 21a9607. * Update FLOPs description (ultralytics#3422) * Update README.md * Changing FLOPS to FLOPs. Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> * Parse URL authentication (ultralytics#3424) * Parse URL authentication * urllib.parse.unquote() * improved error handling * improved error handling * remove %3F * update check_file() * Add FLOPs title to table (ultralytics#3453) * Suppress jit trace warning + graph once (ultralytics#3454) * Suppress jit trace warning + graph once Suppress harmless jit trace warning on TensorBoard add_graph call. Also fix multiple add_graph() calls bug, now only on batch 0. * Update train.py * Update MixUp augmentation `alpha=beta=32.0` (ultralytics#3455) Per VOC empirical results ultralytics#3380 (comment) by @developer0hye * Add `timeout()` class (ultralytics#3460) * Add `timeout()` class * rearrange order * Faster HSV augmentation (ultralytics#3462) remove datatype conversion process that can be skipped * Add `check_git_status()` 5 second timeout (ultralytics#3464) * Add check_git_status() 5 second timeout This should prevent the SSH Git bug that we were discussing @kalenmike * cleanup * replace timeout with check_output built-in timeout * Improved `check_requirements()` offline-handling (ultralytics#3466) Improve robustness of `check_requirements()` function to offline environments (do not attempt pip installs when offline). * Add `output_names` argument for ONNX export with dynamic axes (ultralytics#3456) * Add output names & dynamic axes for onnx export Add output_names and dynamic_axes names for all outputs in torch.onnx.export. The first four outputs of the model will have names output0, output1, output2, output3 * use first output only + cleanup Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Revert FP16 `test.py` and `detect.py` inference to FP32 default (ultralytics#3423) * fixed inference bug ,while use half precision * replace --use-half with --half * replace space and PEP8 in detect.py * PEP8 detect.py * update --half help comment * Update test.py * revert space Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Add additional links/resources to stale.yml message (ultralytics#3467) * Update stale.yml * cleanup * Update stale.yml * reformat * Update stale.yml HUB URL (ultralytics#3468) * Stale `github.actor` bug fix (ultralytics#3483) * Explicit `model.eval()` call `if opt.train=False` (ultralytics#3475) * call model.eval() when opt.train is False call model.eval() when opt.train is False * single-line if statement * cleanup Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * check_requirements() exclude `opencv-python` (ultralytics#3495) Fix for 3rd party or contrib versions of installed OpenCV as in ultralytics#3494. * Earlier `assert` for cpu and half option (ultralytics#3508) * early assert for cpu and half option early assert for cpu and half option * Modified comment Modified comment * Update tutorial.ipynb (ultralytics#3510) * Reduce test.py results spacing (ultralytics#3511) * Update README.md (ultralytics#3512) * Update README.md Minor modifications * 850 width * Update greetings.yml revert greeting change as PRs will now merge to master. Co-authored-by: Piotr Skalski <SkalskiP@users.noreply.github.com> Co-authored-by: SkalskiP <piotr.skalski92@gmail.com> Co-authored-by: Peretz Cohen <pizzaz93@users.noreply.github.com> Co-authored-by: tudoulei <34886368+tudoulei@users.noreply.github.com> Co-authored-by: chocosaj <chocosaj@users.noreply.github.com> Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> Co-authored-by: Yonghye Kwon <developer.0hye@gmail.com> Co-authored-by: Sam_S <SamSamhuns@users.noreply.github.com> Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: edificewang <609552430@qq.com> (cherry picked from commit f3c3d2c)

@kalenmike

* Enable direct `--weights URL` definition @kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ``` * cleanup * bug fixes * weights = attempt_download(weights) * Update experimental.py * Update hubconf.py * return bug fix * comment mirror * min_bytes (cherry picked from commit ba6f3f9)

@kalenmike

* Enable direct `--weights URL` definition @kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ``` * cleanup * bug fixes * weights = attempt_download(weights) * Update experimental.py * Update hubconf.py * return bug fix * comment mirror * min_bytes

@kalenmike

* update ci-testing.yml (ultralytics#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * update ci-testing.yml (ultralytics#3322) * update ci-testing.yml * update greetings.yml * bring back os matrix * Enable direct `--weights URL` definition (ultralytics#3373) * Enable direct `--weights URL` definition @kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ``` * cleanup * bug fixes * weights = attempt_download(weights) * Update experimental.py * Update hubconf.py * return bug fix * comment mirror * min_bytes * Update tutorial.ipynb (ultralytics#3368) add Open in Kaggle badge * `cv2.imread(img, -1)` for IMREAD_UNCHANGED (ultralytics#3379) * Update datasets.py * comment Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * COCO evolution fix (ultralytics#3388) * COCO evolution fix * cleanup * update print * print fix * Create `is_pip()` function (ultralytics#3391) Returns `True` if file is part of pip package. Useful for contextual behavior modification. ```python def is_pip(): # Is file in a pip package? return 'site-packages' in Path(__file__).absolute().parts ``` * Revert "`cv2.imread(img, -1)` for IMREAD_UNCHANGED (ultralytics#3379)" (ultralytics#3395) This reverts commit 67db4b6. * Update FLOPs description (ultralytics#3422) * Update README.md * Changing FLOPS to FLOPs. Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> * Parse URL authentication (ultralytics#3424) * Parse URL authentication * urllib.parse.unquote() * improved error handling * improved error handling * remove %3F * update check_file() * Add FLOPs title to table (ultralytics#3453) * Suppress jit trace warning + graph once (ultralytics#3454) * Suppress jit trace warning + graph once Suppress harmless jit trace warning on TensorBoard add_graph call. Also fix multiple add_graph() calls bug, now only on batch 0. * Update train.py * Update MixUp augmentation `alpha=beta=32.0` (ultralytics#3455) Per VOC empirical results ultralytics#3380 (comment) by @developer0hye * Add `timeout()` class (ultralytics#3460) * Add `timeout()` class * rearrange order * Faster HSV augmentation (ultralytics#3462) remove datatype conversion process that can be skipped * Add `check_git_status()` 5 second timeout (ultralytics#3464) * Add check_git_status() 5 second timeout This should prevent the SSH Git bug that we were discussing @kalenmike * cleanup * replace timeout with check_output built-in timeout * Improved `check_requirements()` offline-handling (ultralytics#3466) Improve robustness of `check_requirements()` function to offline environments (do not attempt pip installs when offline). * Add `output_names` argument for ONNX export with dynamic axes (ultralytics#3456) * Add output names & dynamic axes for onnx export Add output_names and dynamic_axes names for all outputs in torch.onnx.export. The first four outputs of the model will have names output0, output1, output2, output3 * use first output only + cleanup Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Revert FP16 `test.py` and `detect.py` inference to FP32 default (ultralytics#3423) * fixed inference bug ,while use half precision * replace --use-half with --half * replace space and PEP8 in detect.py * PEP8 detect.py * update --half help comment * Update test.py * revert space Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * Add additional links/resources to stale.yml message (ultralytics#3467) * Update stale.yml * cleanup * Update stale.yml * reformat * Update stale.yml HUB URL (ultralytics#3468) * Stale `github.actor` bug fix (ultralytics#3483) * Explicit `model.eval()` call `if opt.train=False` (ultralytics#3475) * call model.eval() when opt.train is False call model.eval() when opt.train is False * single-line if statement * cleanup Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> * check_requirements() exclude `opencv-python` (ultralytics#3495) Fix for 3rd party or contrib versions of installed OpenCV as in ultralytics#3494. * Earlier `assert` for cpu and half option (ultralytics#3508) * early assert for cpu and half option early assert for cpu and half option * Modified comment Modified comment * Update tutorial.ipynb (ultralytics#3510) * Reduce test.py results spacing (ultralytics#3511) * Update README.md (ultralytics#3512) * Update README.md Minor modifications * 850 width * Update greetings.yml revert greeting change as PRs will now merge to master. Co-authored-by: Piotr Skalski <SkalskiP@users.noreply.github.com> Co-authored-by: SkalskiP <piotr.skalski92@gmail.com> Co-authored-by: Peretz Cohen <pizzaz93@users.noreply.github.com> Co-authored-by: tudoulei <34886368+tudoulei@users.noreply.github.com> Co-authored-by: chocosaj <chocosaj@users.noreply.github.com> Co-authored-by: BuildTools <unconfigured@null.spigotmc.org> Co-authored-by: Yonghye Kwon <developer.0hye@gmail.com> Co-authored-by: Sam_S <SamSamhuns@users.noreply.github.com> Co-authored-by: Samridha Shrestha <samridha.shrestha@g42.ai> Co-authored-by: edificewang <609552430@qq.com>

Enable direct --weights URL definition

f65ec28

@kalenmike this PR will enable direct --weights URL definition. Example use case: ``` python train.py --weights https://storage.googleapis.com/bucket/dir/model.pt ```

glenn-jocher added 3 commits May 28, 2021 13:35

cleanup

1a475fb

bug fixes

e383f52

weights = attempt_download(weights)

0b13b94

glenn-jocher self-assigned this May 28, 2021

glenn-jocher added 5 commits May 28, 2021 14:24

Update experimental.py

d588968

Update hubconf.py

eca773d

return bug fix

d9e5c9a

comment mirror

0cd14c4

min_bytes

eecf1c7

glenn-jocher merged commit ba6f3f9 into develop May 28, 2021

glenn-jocher deleted the glenn-jocher-patch-2 branch May 28, 2021 13:18

glenn-jocher mentioned this pull request Jun 2, 2021

Parse URL authentication #3424

Merged

glenn-jocher mentioned this pull request Oct 12, 2021

YOLOv5 release v6.0 #5141

Merged

glenn-jocher mentioned this pull request Nov 7, 2021

YOLOv5 v6.0 compatibility update (draft) ultralytics/yolov3#1855

Closed

glenn-jocher mentioned this pull request Nov 14, 2021

YOLOv5 v6.0 compatibility update ultralytics/yolov3#1857

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable direct `--weights URL` definition #3373

Enable direct `--weights URL` definition #3373

glenn-jocher commented May 28, 2021 •

edited by UltralyticsAssistant

Loading

glenn-jocher commented May 28, 2021 •

edited

Loading

glenn-jocher commented May 28, 2021

Enable direct --weights URL definition #3373

Enable direct --weights URL definition #3373

Conversation

glenn-jocher commented May 28, 2021 • edited by UltralyticsAssistant Loading

🛠️ PR Summary

🌟 Summary

📊 Key Changes

🎯 Purpose & Impact

glenn-jocher commented May 28, 2021 • edited Loading

glenn-jocher commented May 28, 2021

Enable direct `--weights URL` definition #3373

Enable direct `--weights URL` definition #3373

glenn-jocher commented May 28, 2021 •

edited by UltralyticsAssistant

Loading

glenn-jocher commented May 28, 2021 •

edited

Loading