📃Update documentation (#280)

* Add initial draft

* Add wandb sweep image

* Refactor logging graph

* Modify callbacks init for new design

* Update images to hazelnut dataset

* Add unwatch to graphlogger callback

* Update text to match new design

* Ignore mypy error

* Replace paroject params

* Fix project path

* Fix visualizer test

* Address PR comments + markdown->rst

* Fix torchmetrics version

* Fix tests

* Add metrics configuration callback to benchmarking

* Change wandb_sweep to sweep

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

README.md

@@ -189,12 +189,48 @@ python tools/inference.py \
 
 > Ensure that you provide path to `meta_data.json` if you want the normalization to be applied correctly.
 
+## Hyperparameter Optimization
+
+To run hyperparameter optimization, use the following command:
+
+```bash
+python tools/hpo/sweep.py \
+    --model padim --model_config ./path_to_config.yaml \
+    --sweep_config tools/hpo/sweep.yaml
+```
+
+For more details refer the [HPO Documentation](https://openvinotoolkit.github.io/anomalib/guides/hyperparameter_optimization.html)
+
+## Benchmarking
+
+To gather benchmarking data such as throughput across categories, use the following command:
+
+```bash
+python tools/benchmarking/benchmark.py \
+    --config <relative/absolute path>/<paramfile>.yaml
+```
+
+Refer to the [Benchmarking Documentation](https://openvinotoolkit.github.io/anomalib/guides/benchmarking.html) for more details.
+
+## Logging Images
+
+You can save images locally or to a logger such TensorBoard or Weights and Biases by setting the following configuration.
+
+```yaml
+logging:
+  logger: [tensorboard, wandb]
+  log_graph: false
+```
+
+For more information on logging images, refer to the [Logging Documentation](https://openvinotoolkit.github.io/anomalib/guides/logging.html)
 ___
 
 ## Datasets
+
 `anomalib` supports MVTec AD [(CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/) and BeanTech [(CC-BY-SA)](https://creativecommons.org/licenses/by-sa/4.0/legalcode) for benchmarking and `folder` for custom dataset training/inference.
 
 ### [MVTec AD Dataset](https://www.mvtec.com/company/research/datasets/mvtec-ad)
+
 MVTec AD dataset is one of the main benchmarks for anomaly detection, and is released under the
 Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License [(CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/).
 

anomalib/models/cflow/config.yaml

@@ -49,8 +49,11 @@ metrics:
 project:
   seed: 0
   path: ./results
-  log_images_to: [local]
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 # PL Trainer Args. Don't add extra parameter here.
 trainer:

anomalib/models/dfkde/config.yaml

@@ -35,8 +35,11 @@ metrics:
 project:
   seed: 42
   path: ./results
-  log_images_to: []
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 # PL Trainer Args. Don't add extra parameter here.
 trainer:

anomalib/models/dfm/config.yaml

@@ -37,8 +37,11 @@ metrics:
 project:
   seed: 42
   path: ./results
-  log_images_to: []
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 # PL Trainer Args. Don't add extra parameter here.
 trainer:

anomalib/models/ganomaly/config.yaml

@@ -37,6 +37,7 @@ model:
   wadv: 1
   wcon: 50
   wenc: 1
+  normalization_method: none
 
 metrics:
   image:
@@ -52,8 +53,11 @@ metrics:
 project:
   seed: 0
   path: ./results
-  log_images_to: []
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 optimization:
   openvino:

anomalib/models/padim/config.yaml

@@ -44,8 +44,11 @@ metrics:
 project:
   seed: 42
   path: ./results
-  log_images_to: ["local"]
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 optimization:
   openvino:

anomalib/models/patchcore/config.yaml

@@ -46,8 +46,11 @@ metrics:
 project:
   seed: 0
   path: ./results
-  log_images_to: [local]
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 # PL Trainer Args. Don't add extra parameter here.
 trainer:

anomalib/models/stfpm/config.yaml

@@ -52,8 +52,11 @@ metrics:
 project:
   seed: 0
   path: ./results
-  log_images_to: [local]
-  logger: false # options: [tensorboard, wandb, csv] or combinations.
+
+logging:
+  log_images_to: ["local"] # options: [wandb, tensorboard, local]. Make sure you also set logger with using wandb or tensorboard.
+  logger: [] # options: [tensorboard, wandb, csv] or combinations.
+  log_graph: false # Logs the model graph to respective logger.
 
 optimization:
   openvino:

anomalib/utils/callbacks/__init__.py

@@ -15,6 +15,7 @@
 # and limitations under the License.
 
 import os
+import warnings
 from importlib import import_module
 from typing import List, Union
 
@@ -23,6 +24,7 @@
 from pytorch_lightning.callbacks import Callback, ModelCheckpoint
 
 from .cdf_normalization import CdfNormalizationCallback
+from .graph import GraphLogger
 from .metrics_configuration import MetricsConfigurationCallback
 from .min_max_normalization import MinMaxNormalizationCallback
 from .model_loader import LoadModelCallback
@@ -98,11 +100,20 @@ def get_callbacks(config: Union[ListConfig, DictConfig]) -> List[Callback]:
         else:
             raise ValueError(f"Normalization method not recognized: {config.model.normalization_method}")
 
-    if not config.project.log_images_to == []:
+    # TODO Modify when logger is deprecated from project
+    if "log_images_to" in config.project.keys():
+        warnings.warn(
+            "'log_images_to' key will be deprecated from 'project' section of the config file."
+            " Please use the logging section in config file",
+            DeprecationWarning,
+        )
+        config.logging.log_images_to = config.project.log_images_to
+
+    if not config.logging.log_images_to == []:
         callbacks.append(
             VisualizerCallback(
                 task=config.dataset.task,
-                log_images_to=config.project.log_images_to,
+                log_images_to=config.logging.log_images_to,
                 inputs_are_normalized=not config.model.normalization_method == "none",
             )
         )
@@ -133,4 +144,8 @@ def get_callbacks(config: Union[ListConfig, DictConfig]) -> List[Callback]:
                 )
             )
 
+    # Add callback to log graph to loggers
+    if config.logging.log_graph not in [None, False]:
+        callbacks.append(GraphLogger())
+
     return callbacks

anomalib/utils/callbacks/graph.py

@@ -0,0 +1,53 @@
+"""Log model graph to respective logger."""
+
+# Copyright (C) 2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+
+import torch
+from pytorch_lightning import Callback, LightningModule, Trainer
+
+from anomalib.utils.loggers import AnomalibTensorBoardLogger, AnomalibWandbLogger
+
+
+class GraphLogger(Callback):
+    """Log model graph to respective logger."""
+
+    def on_train_start(self, trainer: Trainer, pl_module: LightningModule) -> None:
+        """Log model graph to respective logger.
+
+        Args:
+            trainer: Trainer object which contans reference to loggers.
+            pl_module: LightningModule object which is logged.
+        """
+
+        for logger in trainer.loggers:
+            if isinstance(logger, AnomalibWandbLogger):
+                # NOTE: log graph gets populated only after one backward pass. This won't work for models which do not
+                # require training such as Padim
+                logger.watch(pl_module, log_graph=True, log="all")
+                break
+
+    def on_train_end(self, trainer: Trainer, pl_module: LightningModule) -> None:
+        """Unwatch model if configured for wandb and log it model graph in Tensorboard if specified.
+
+        Args:
+            trainer: Trainer object which contans reference to loggers.
+            pl_module: LightningModule object which is logged.
+        """
+
+        for logger in trainer.loggers:
+            if isinstance(logger, AnomalibTensorBoardLogger):
+                logger.log_graph(pl_module, input_array=torch.ones((1, 3, 256, 256)))
+            elif isinstance(logger, AnomalibWandbLogger):
+                logger.unwatch(pl_module)  # type: ignore

anomalib/utils/loggers/__init__.py

@@ -16,6 +16,7 @@
 
 import logging
 import os
+import warnings
 from typing import Iterable, List, Union
 
 from omegaconf.dictconfig import DictConfig
@@ -74,36 +75,55 @@ def get_experiment_logger(
     Returns:
         Union[LightningLoggerBase, Iterable[LightningLoggerBase], bool]: Logger
     """
-    if config.project.logger in [None, False]:
+
+    # TODO remove when logger is deprecated from project
+    if "logger" in config.project.keys():
+        warnings.warn(
+            "'logger' key will be deprecated from 'project' section of the config file."
+            " Please use the logging section in config file.",
+            DeprecationWarning,
+        )
+        if "logging" not in config:
+            config.logging = {"logger": config.project.logger, "log_graph": False}
+        else:
+            config.logging.logger = config.project.logger
+
+    if config.logging.logger in [None, False]:
         return False
 
     logger_list: List[LightningLoggerBase] = []
-    if isinstance(config.project.logger, str):
-        config.project.logger = [config.project.logger]
+    if isinstance(config.logging.logger, str):
+        config.logging.logger = [config.logging.logger]
 
-    for logger in config.project.logger:
+    for logger in config.logging.logger:
         if logger == "tensorboard":
             logger_list.append(
                 AnomalibTensorBoardLogger(
                     name="Tensorboard Logs",
                     save_dir=os.path.join(config.project.path, "logs"),
+                    log_graph=config.logging.log_graph,
                 )
             )
         elif logger == "wandb":
             wandb_logdir = os.path.join(config.project.path, "logs")
             os.makedirs(wandb_logdir, exist_ok=True)
+            name = (
+                config.model.name
+                if "category" not in config.dataset.keys()
+                else f"{config.dataset.category} {config.model.name}"
+            )
             logger_list.append(
                 AnomalibWandbLogger(
                     project=config.dataset.name,
-                    name=f"{config.dataset.category} {config.model.name}",
+                    name=name,
                     save_dir=wandb_logdir,
                 )
             )
         elif logger == "csv":
             logger_list.append(CSVLogger(save_dir=os.path.join(config.project.path, "logs")))
         else:
             raise UnknownLogger(
-                f"Unknown logger type: {config.project.logger}. "
+                f"Unknown logger type: {config.logging.logger}. "
                 f"Available loggers are: {AVAILABLE_LOGGERS}.\n"
                 f"To enable the logger, set `project.logger` to `true` or use one of available loggers in config.yaml\n"
                 f"To disable the logger, set `project.logger` to `false`."

docs/source/conf.py

@@ -73,6 +73,8 @@
     "show-inheritance": True,
 }
 
+myst_enable_extensions = ["colon_fence"]
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 

docs/source/guides/benchmarking.rst

@@ -0,0 +1,55 @@
+.. _benchmarking:
+
+Benchmarking
+=============
+
+To add to the suit of experiment tracking and optimization, anomalib also includes a benchmarking script for gathering results across different combinations of models, their parameters, and dataset categories. The model performance and throughputs are logged into a csv file that can also serve as a means to track model drift. Optionally, these same results can be logged to Weights and Biases and TensorBoard. A sample configuration file is shown below.
+
+.. code-block:: yaml
+
+  seed: 42
+  compute_openvino: false
+  hardware:
+    - cpu
+    - gpu
+  writer:
+    - wandb
+    - tensorboard
+  grid_search:
+    dataset:
+      category:
+        - colour
+        - crack
+      image_size: [128, 256]
+    model_name:
+      - padim
+      - stfpm
+
+This configuration computes the throughput and performance metrics on CPU and GPU for two categories of a custom folder dataset for Padim and STFPM models. To configure a custom dataset, use the respective model configuration file. An example for dataset configuration used in this guide is shown below. Refer `README <https://github.com/openvinotoolkit/anomalib#readme>`_ for more details.
+
+.. code-block:: yaml
+
+  dataset:
+    name: hazelnut
+    format: folder
+    path: path/hazelnut_toy
+    normal_dir: good # name of the folder containing normal images.
+    abnormal_dir: colour # name of the folder containing abnormal images.
+    normal_test_dir: null
+    task: segmentation # classification or segmentation
+    mask: path/hazelnut_toy/mask/colour
+    extensions: .jpg
+    split_ratio: 0.2
+    seed: 0
+    image_size: 256
+
+By default, ``compute_openvino`` is set to ``False`` to support instances where OpenVINO requirements are not installed in the environment. Once installed, this flag can be set to ``True`` to get the throughput on OpenVINO optimized models. The ``writer`` parameter is optional and can be set to ``writer: []`` in case the user only requires a csv file without logging to each respective logger. It is a good practice to set a value of seed to ensure reproducibility across runs and thus, is set to a non-zero value by default.
+
+Once a configuration is decided, benchmarking can easily be performed by calling
+
+.. code-block:: bash
+
+  python tools/benchmarking/benchmark.py --config <relative/absolute path>/<paramfile>.yaml
+
+
+A nice feature about the provided benchmarking script is that if the host system has multiple GPUs, the runs are parallelized over all the available GPUs for faster collection of result.