Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Add notebook for hpo #592

Merged
merged 2 commits into from
Sep 28, 2022
Merged

✨ Add notebook for hpo #592

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
449d98d
Add notebook for hpo
Sep 27, 2022
bf9d96d
Reference notebook in docs
Sep 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/how_to_guides/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ How to Guides
notebooks/100_datamodules/103_folder.ipynb
notebooks/200_models/201_fastflow.ipynb
notebooks/300_benchmarking/301_benchmarking.ipynb
notebooks/300_benchmarking/302_hpo_wandb.ipynb
389 changes: 389 additions & 0 deletions notebooks/300_benchmarking/302_hpo_wandb.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,389 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "owTxSDKzB9oO"
},
"source": [
"# Walkthrough on Hyperparameter Optimization using Weights and Biases"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z3pB7jatCCxt"
},
"source": [
"## Clone and Install Anomalib"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "N8xxNHt8B3yZ",
"outputId": "7cd2713f-2bd2-4279-a600-e85d74e4db73"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'anomalib'...\n",
"remote: Enumerating objects: 21618, done.\u001b[K\n",
"remote: Counting objects: 100% (3157/3157), done.\u001b[K\n",
"remote: Compressing objects: 100% (1047/1047), done.\u001b[K\n",
"remote: Total 21618 (delta 1785), reused 2905 (delta 1622), pack-reused 18461\u001b[K\n",
"Receiving objects: 100% (21618/21618), 49.64 MiB | 35.03 MiB/s, done.\n",
"Resolving deltas: 100% (12658/12658), done.\n"
]
}
],
"source": [
"!git clone https://github.com/openvinotoolkit/anomalib.git"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "yT-ZpzFXCGrU",
"outputId": "dcff1445-1b3b-42ca-de03-176c61a03127"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/content/anomalib\n"
]
}
],
"source": [
"%cd anomalib"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "w72Q-XRmCKau"
},
"outputs": [],
"source": [
"%pip install ."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BkrI2pI5C6Z-"
},
"source": [
"> Note: Restart Runtime if promted by clicking the button at the end of the install logs"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gsWKy1FUCXoQ"
},
"source": [
"## Download and Setup Dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4l_z-KK0C98t",
"outputId": "1b2dab18-5fcf-406e-d6e0-1ceab4e10c22"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/content/anomalib\n"
]
}
],
"source": [
"%cd /content/anomalib/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7E8nMjLVCXXg"
},
"outputs": [],
"source": [
"!wget https://openvinotoolkit.github.io/anomalib/_downloads/3f2af1d7748194b18c2177a34c03a2c4/hazelnut_toy.zip"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "EaQ5BOS5DCA_"
},
"outputs": [],
"source": [
"!mkdir datasets && unzip hazelnut_toy.zip -d datasets/ > /dev/null"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Zhc8ai8VGKGZ"
},
"source": [
"## Creating training configuration\n",
"\n",
"Since we are using our [custom dataset](https://openvinotoolkit.github.io/anomalib/how_to_guides/train_custom_data.html) we need to modify the default configuration file. The following configuration is based on the one available here `anomalib/anomalib/models/stfpm/config.yaml`"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "ZqhdaJElGkCi"
},
"outputs": [],
"source": [
"folder_stfpm = \"\"\"dataset:\n",
" name: hazelnut\n",
" format: folder\n",
" path: /content/anomalib/datasets/hazelnut_toy\n",
" normal_dir: good # name of the folder containing normal images.\n",
" abnormal_dir: colour # name of the folder containing abnormal images.\n",
" normal_test_dir: null # name of the folder containing normal test images.\n",
" task: segmentation # classification or segmentation\n",
" mask: /content/anomalib/datasets/hazelnut_toy/mask/colour # optional\n",
" extensions: null\n",
" split_ratio: 0.2 # ratio of the normal images that will be used to create a test split\n",
" image_size: 256\n",
" train_batch_size: 32\n",
" test_batch_size: 32\n",
" num_workers: 8\n",
" transform_config:\n",
" train: null\n",
" val: null\n",
" create_validation_set: false\n",
" tiling:\n",
" apply: false\n",
" tile_size: null\n",
" stride: null\n",
" remove_border_count: 0\n",
" use_random_tiling: False\n",
" random_tile_count: 16\n",
"\n",
"model:\n",
" name: stfpm\n",
" backbone: resnet18\n",
" layers:\n",
" - layer1\n",
" - layer2\n",
" - layer3\n",
" lr: 0.4\n",
" momentum: 0.9\n",
" weight_decay: 0.0001\n",
" early_stopping:\n",
" patience: 3\n",
" metric: pixel_AUROC\n",
" mode: max\n",
" normalization_method: min_max # options: [null, min_max, cdf]\n",
"\n",
"metrics:\n",
" image:\n",
" - F1Score\n",
" - AUROC\n",
" pixel:\n",
" - F1Score\n",
" - AUROC\n",
" threshold:\n",
" image_default: 0\n",
" pixel_default: 0\n",
" adaptive: true\n",
"\n",
"visualization:\n",
" show_images: False # show images on the screen\n",
" save_images: False # save images to the file system\n",
" log_images: False # log images to the available loggers (if any)\n",
" image_save_path: null # path to which images will be saved\n",
" mode: full # options: [\"full\", \"simple\"]\n",
"\n",
"project:\n",
" seed: 0\n",
" path: ./results\n",
"\n",
"logging:\n",
" logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.\n",
" log_graph: false # Logs the model graph to respective logger.\n",
"\n",
"optimization:\n",
" export_mode: null #options: onnx, openvino\n",
"# PL Trainer Args. Don't add extra parameter here.\n",
"trainer:\n",
" accelerator: auto # <\"cpu\", \"gpu\", \"tpu\", \"ipu\", \"hpu\", \"auto\">\n",
" accumulate_grad_batches: 1\n",
" amp_backend: native\n",
" auto_lr_find: false\n",
" auto_scale_batch_size: false\n",
" auto_select_gpus: false\n",
" benchmark: false\n",
" check_val_every_n_epoch: 1\n",
" default_root_dir: null\n",
" detect_anomaly: false\n",
" deterministic: false\n",
" devices: 1\n",
" enable_checkpointing: true\n",
" enable_model_summary: true\n",
" enable_progress_bar: true\n",
" fast_dev_run: false\n",
" gpus: null # Set automatically\n",
" gradient_clip_val: 0\n",
" ipus: null\n",
" limit_predict_batches: 1.0\n",
" limit_test_batches: 1.0\n",
" limit_train_batches: 1.0\n",
" limit_val_batches: 1.0\n",
" log_every_n_steps: 50\n",
" log_gpu_memory: null\n",
" max_epochs: 100\n",
" max_steps: -1\n",
" max_time: null\n",
" min_epochs: null\n",
" min_steps: null\n",
" move_metrics_to_cpu: false\n",
" multiple_trainloader_mode: max_size_cycle\n",
" num_nodes: 1\n",
" num_processes: null\n",
" num_sanity_val_steps: 0\n",
" overfit_batches: 0.0\n",
" plugins: null\n",
" precision: 32\n",
" profiler: null\n",
" reload_dataloaders_every_n_epochs: 0\n",
" replace_sampler_ddp: true\n",
" strategy: null\n",
" sync_batchnorm: false\n",
" tpu_cores: null\n",
" track_grad_norm: -1\n",
" val_check_interval: 1.0\n",
"\"\"\"\n",
"\n",
"with open(\"model_config.yaml\", \"w\", encoding=\"utf8\") as f:\n",
" f.writelines(folder_stfpm)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R7broXB-EYnx"
},
"source": [
"## Create sweep configuration\n",
"\n",
"The following configuration file is based on the one available at `anomalib/tools/hpo/configs/stfpm.yaml`. For this example we will use the STFPM model."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"id": "D-7YuDU3G_tG"
},
"outputs": [],
"source": [
"sweep_config = \"\"\"observation_budget: 10\n",
"method: bayes\n",
"metric:\n",
" name: pixel_AUROC\n",
" goal: maximize\n",
"parameters:\n",
" dataset:\n",
" category: capsule\n",
" image_size:\n",
" values: [128, 256]\n",
" model:\n",
" backbone:\n",
" values: [resnet18, wide_resnet50_2]\n",
" lr:\n",
" min: 0.1\n",
" max: 0.9\n",
" momentum:\n",
" min: 0.1\n",
" max: 0.9\n",
"\"\"\"\n",
"\n",
"with open(\"sweep_config.yaml\", \"w\", encoding=\"utf8\") as f:\n",
" f.writelines(sweep_config)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZVM8GvxUDCfV"
},
"outputs": [],
"source": [
"!python ./tools/hpo/sweep.py --model stfpm --model_config model_config.yaml --sweep_config sweep_config.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OhS88SuxFFzi"
},
"source": [
"While only a few parameters were shown in this example, you can call HPO on any of the parameters defined in the `model` and `dataset` section of the model configuration file."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FX57TV8gFirU"
},
"source": [
"Congratulations 🎉 You have now successfully optimized a model on your dataset.\n",
"\n",
"To go into more detail, refer our documentation on [hyperparameter optimization](https://openvinotoolkit.github.io/anomalib/tutorials/hyperparameter_optimization.html)."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3.8.0 ('anomalib')",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.8.0"
},
"vscode": {
"interpreter": {
"hash": "ba723bee1893023fba5911c5ba85dac05fe2496fa0862b3e274bad096c0e1e2a"
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Loading