📝 📊 Add notebook for hpo (#592)

* Add notebook for hpo

* Reference notebook in docs

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

docs/source/how_to_guides/index.rst

@@ -12,3 +12,4 @@ How to Guides
    notebooks/100_datamodules/103_folder.ipynb
    notebooks/200_models/201_fastflow.ipynb
    notebooks/300_benchmarking/301_benchmarking.ipynb
+   notebooks/300_benchmarking/302_hpo_wandb.ipynb

notebooks/300_benchmarking/302_hpo_wandb.ipynb

@@ -0,0 +1,389 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "owTxSDKzB9oO"
+      },
+      "source": [
+        "# Walkthrough on Hyperparameter Optimization using Weights and Biases"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "z3pB7jatCCxt"
+      },
+      "source": [
+        "## Clone and Install Anomalib"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "N8xxNHt8B3yZ",
+        "outputId": "7cd2713f-2bd2-4279-a600-e85d74e4db73"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Cloning into 'anomalib'...\n",
+            "remote: Enumerating objects: 21618, done.\u001b[K\n",
+            "remote: Counting objects: 100% (3157/3157), done.\u001b[K\n",
+            "remote: Compressing objects: 100% (1047/1047), done.\u001b[K\n",
+            "remote: Total 21618 (delta 1785), reused 2905 (delta 1622), pack-reused 18461\u001b[K\n",
+            "Receiving objects: 100% (21618/21618), 49.64 MiB | 35.03 MiB/s, done.\n",
+            "Resolving deltas: 100% (12658/12658), done.\n"
+          ]
+        }
+      ],
+      "source": [
+        "!git clone https://github.com/openvinotoolkit/anomalib.git"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "yT-ZpzFXCGrU",
+        "outputId": "dcff1445-1b3b-42ca-de03-176c61a03127"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "/content/anomalib\n"
+          ]
+        }
+      ],
+      "source": [
+        "%cd anomalib"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "w72Q-XRmCKau"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install ."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BkrI2pI5C6Z-"
+      },
+      "source": [
+        "> Note: Restart Runtime if promted by clicking the button at the end of the install logs"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gsWKy1FUCXoQ"
+      },
+      "source": [
+        "## Download and Setup Dataset"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "4l_z-KK0C98t",
+        "outputId": "1b2dab18-5fcf-406e-d6e0-1ceab4e10c22"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "/content/anomalib\n"
+          ]
+        }
+      ],
+      "source": [
+        "%cd /content/anomalib/"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7E8nMjLVCXXg"
+      },
+      "outputs": [],
+      "source": [
+        "!wget https://openvinotoolkit.github.io/anomalib/_downloads/3f2af1d7748194b18c2177a34c03a2c4/hazelnut_toy.zip"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "id": "EaQ5BOS5DCA_"
+      },
+      "outputs": [],
+      "source": [
+        "!mkdir datasets && unzip hazelnut_toy.zip -d datasets/ > /dev/null"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Zhc8ai8VGKGZ"
+      },
+      "source": [
+        "## Creating training configuration\n",
+        "\n",
+        "Since we are using our [custom dataset](https://openvinotoolkit.github.io/anomalib/how_to_guides/train_custom_data.html) we need to modify the default configuration file. The following configuration is based on the one available here `anomalib/anomalib/models/stfpm/config.yaml`"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "metadata": {
+        "id": "ZqhdaJElGkCi"
+      },
+      "outputs": [],
+      "source": [
+        "folder_stfpm = \"\"\"dataset:\n",
+        "  name: hazelnut\n",
+        "  format: folder\n",
+        "  path: /content/anomalib/datasets/hazelnut_toy\n",
+        "  normal_dir: good # name of the folder containing normal images.\n",
+        "  abnormal_dir: colour # name of the folder containing abnormal images.\n",
+        "  normal_test_dir: null # name of the folder containing normal test images.\n",
+        "  task: segmentation # classification or segmentation\n",
+        "  mask: /content/anomalib/datasets/hazelnut_toy/mask/colour # optional\n",
+        "  extensions: null\n",
+        "  split_ratio: 0.2  # ratio of the normal images that will be used to create a test split\n",
+        "  image_size: 256\n",
+        "  train_batch_size: 32\n",
+        "  test_batch_size: 32\n",
+        "  num_workers: 8\n",
+        "  transform_config:\n",
+        "    train: null\n",
+        "    val: null\n",
+        "  create_validation_set: false\n",
+        "  tiling:\n",
+        "    apply: false\n",
+        "    tile_size: null\n",
+        "    stride: null\n",
+        "    remove_border_count: 0\n",
+        "    use_random_tiling: False\n",
+        "    random_tile_count: 16\n",
+        "\n",
+        "model:\n",
+        "  name: stfpm\n",
+        "  backbone: resnet18\n",
+        "  layers:\n",
+        "    - layer1\n",
+        "    - layer2\n",
+        "    - layer3\n",
+        "  lr: 0.4\n",
+        "  momentum: 0.9\n",
+        "  weight_decay: 0.0001\n",
+        "  early_stopping:\n",
+        "    patience: 3\n",
+        "    metric: pixel_AUROC\n",
+        "    mode: max\n",
+        "  normalization_method: min_max # options: [null, min_max, cdf]\n",
+        "\n",
+        "metrics:\n",
+        "  image:\n",
+        "    - F1Score\n",
+        "    - AUROC\n",
+        "  pixel:\n",
+        "    - F1Score\n",
+        "    - AUROC\n",
+        "  threshold:\n",
+        "    image_default: 0\n",
+        "    pixel_default: 0\n",
+        "    adaptive: true\n",
+        "\n",
+        "visualization:\n",
+        "  show_images: False # show images on the screen\n",
+        "  save_images: False # save images to the file system\n",
+        "  log_images: False # log images to the available loggers (if any)\n",
+        "  image_save_path: null # path to which images will be saved\n",
+        "  mode: full # options: [\"full\", \"simple\"]\n",
+        "\n",
+        "project:\n",
+        "  seed: 0\n",
+        "  path: ./results\n",
+        "\n",
+        "logging:\n",
+        "  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.\n",
+        "  log_graph: false # Logs the model graph to respective logger.\n",
+        "\n",
+        "optimization:\n",
+        "  export_mode: null #options: onnx, openvino\n",
+        "# PL Trainer Args. Don't add extra parameter here.\n",
+        "trainer:\n",
+        "  accelerator: auto # <\"cpu\", \"gpu\", \"tpu\", \"ipu\", \"hpu\", \"auto\">\n",
+        "  accumulate_grad_batches: 1\n",
+        "  amp_backend: native\n",
+        "  auto_lr_find: false\n",
+        "  auto_scale_batch_size: false\n",
+        "  auto_select_gpus: false\n",
+        "  benchmark: false\n",
+        "  check_val_every_n_epoch: 1\n",
+        "  default_root_dir: null\n",
+        "  detect_anomaly: false\n",
+        "  deterministic: false\n",
+        "  devices: 1\n",
+        "  enable_checkpointing: true\n",
+        "  enable_model_summary: true\n",
+        "  enable_progress_bar: true\n",
+        "  fast_dev_run: false\n",
+        "  gpus: null # Set automatically\n",
+        "  gradient_clip_val: 0\n",
+        "  ipus: null\n",
+        "  limit_predict_batches: 1.0\n",
+        "  limit_test_batches: 1.0\n",
+        "  limit_train_batches: 1.0\n",
+        "  limit_val_batches: 1.0\n",
+        "  log_every_n_steps: 50\n",
+        "  log_gpu_memory: null\n",
+        "  max_epochs: 100\n",
+        "  max_steps: -1\n",
+        "  max_time: null\n",
+        "  min_epochs: null\n",
+        "  min_steps: null\n",
+        "  move_metrics_to_cpu: false\n",
+        "  multiple_trainloader_mode: max_size_cycle\n",
+        "  num_nodes: 1\n",
+        "  num_processes: null\n",
+        "  num_sanity_val_steps: 0\n",
+        "  overfit_batches: 0.0\n",
+        "  plugins: null\n",
+        "  precision: 32\n",
+        "  profiler: null\n",
+        "  reload_dataloaders_every_n_epochs: 0\n",
+        "  replace_sampler_ddp: true\n",
+        "  strategy: null\n",
+        "  sync_batchnorm: false\n",
+        "  tpu_cores: null\n",
+        "  track_grad_norm: -1\n",
+        "  val_check_interval: 1.0\n",
+        "\"\"\"\n",
+        "\n",
+        "with open(\"model_config.yaml\", \"w\", encoding=\"utf8\") as f:\n",
+        "    f.writelines(folder_stfpm)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "R7broXB-EYnx"
+      },
+      "source": [
+        "## Create sweep configuration\n",
+        "\n",
+        "The following configuration file is based on the one available at `anomalib/tools/hpo/configs/stfpm.yaml`. For this example we will use the STFPM model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 32,
+      "metadata": {
+        "id": "D-7YuDU3G_tG"
+      },
+      "outputs": [],
+      "source": [
+        "sweep_config = \"\"\"observation_budget: 10\n",
+        "method: bayes\n",
+        "metric:\n",
+        "  name: pixel_AUROC\n",
+        "  goal: maximize\n",
+        "parameters:\n",
+        "  dataset:\n",
+        "    category: capsule\n",
+        "    image_size:\n",
+        "      values: [128, 256]\n",
+        "  model:\n",
+        "    backbone:\n",
+        "      values: [resnet18, wide_resnet50_2]\n",
+        "    lr:\n",
+        "      min: 0.1\n",
+        "      max: 0.9\n",
+        "    momentum:\n",
+        "      min: 0.1\n",
+        "      max: 0.9\n",
+        "\"\"\"\n",
+        "\n",
+        "with open(\"sweep_config.yaml\", \"w\", encoding=\"utf8\") as f:\n",
+        "    f.writelines(sweep_config)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZVM8GvxUDCfV"
+      },
+      "outputs": [],
+      "source": [
+        "!python ./tools/hpo/sweep.py --model stfpm --model_config model_config.yaml --sweep_config sweep_config.yaml"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "OhS88SuxFFzi"
+      },
+      "source": [
+        "While only a few parameters were shown in this example, you can call HPO on any of the parameters defined in the `model` and `dataset` section of the model configuration file."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FX57TV8gFirU"
+      },
+      "source": [
+        "Congratulations 🎉 You have now successfully optimized a model on your dataset.\n",
+        "\n",
+        "To go into more detail, refer our documentation on [hyperparameter optimization](https://openvinotoolkit.github.io/anomalib/tutorials/hyperparameter_optimization.html)."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3.8.0 ('anomalib')",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.8.0"
+    },
+    "vscode": {
+      "interpreter": {
+        "hash": "ba723bee1893023fba5911c5ba85dac05fe2496fa0862b3e274bad096c0e1e2a"
+      }
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}