📚 Add Pipelines Documentation

docs/source/markdown/guides/reference/models/image/index.md

@@ -114,6 +114,8 @@ Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection
 :link-type: doc
 
 U-Flow: A U-shaped Normalizing Flow for Anomaly Detection with Unsupervised Threshold
+:::
+
 :::{grid-item-card} {material-regular}`model_training;1.5em` WinCLIP
 :link: ./winclip
 :link-type: doc

docs/source/markdown/guides/reference/pipelines/base/generator.md

@@ -0,0 +1,9 @@
+# Generator
+
+```{eval-rst}
+.. autoclass:: anomalib.pipelines.components.base.job.JobGenerator
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+```

docs/source/markdown/guides/reference/pipelines/base/job.md

@@ -0,0 +1,9 @@
+# Job
+
+```{eval-rst}
+.. autoclass:: anomalib.pipelines.components.base.job.Job
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+```

docs/source/markdown/guides/reference/pipelines/base/pipeline.md

@@ -0,0 +1,9 @@
+# Pipeline Base Class
+
+The `Pipeline` class is the base class for all pipelines. It provides the following methods:
+
+```{eval-rst}
+.. automodule:: anomalib.pipelines.components.base.pipeline
+    :members:
+    :show-inheritance:
+```

docs/source/markdown/guides/reference/pipelines/benchmark/generator.md

@@ -0,0 +1,10 @@
+# Benchmark Job Generator
+
+```{eval-rst}
+
+.. autoclass:: anomalib.pipelines.benchmark.generator.BenchmarkJobGenerator
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+```

docs/source/markdown/guides/reference/pipelines/benchmark/index.md

@@ -0,0 +1,91 @@
+# Benchmarking Pipeline
+
+The benchmarking pipeline allows you to run multiple models across combination of parameters and dataset categories to collect metrics. The benchmarking run is configured using a config file that specifies the grid-search parameters. A sample config file is shown below:
+
+```yaml
+accelerator:
+  - cuda
+  - cpu
+benchmark:
+  seed: 42
+  model:
+    class_path:
+      grid_search: [Padim, Patchcore]
+  data:
+    class_path: MVTec
+    init_args:
+      category:
+        grid:
+          - bottle
+          - cable
+          - capsule
+```
+
+The `accelerator` parameter is specific to the pipeline and is used to configure the runners. When `cuda` is passed it adds a [parallel](../runners/parallel.md) runner with number of jobs equal to the number of cuda devices. The idea is that since job is independent, we can increase the throughput by distributing each on an individual accelerator. The `cpu` jobs are run [serially](../runners/serial.md).
+
+## Running the Benchmark Pipeline
+
+There are two ways to run the benchmark pipeline; as a subcommand, or as a standalone entrypoint.
+
+:::::{dropdown} CLI
+:icon: code
+
+::::{tab-set}
+:::{tab-item} Anomalib subcommand
+:sync: label-1
+
+```{literalinclude} ../../../../../snippets/pipelines/benchmark/cli_anomalib.txt
+:language: bash
+```
+
+:::
+
+:::{tab-item} Standalone entrypoint
+:sync: label-2
+
+```{literalinclude} ../../../../../snippets/pipelines/benchmark/cli_tools.txt
+:language: bash
+```
+
+:::
+
+:::::
+
+## Benchmark Pipeline Class
+
+```{eval-rst}
+
+.. autoclass:: anomalib.pipelines.benchmark.pipeline.Benchmark
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+```
+
+::::{grid}
+:margin: 1 1 0 0
+:gutter: 1
+
+:::{grid-item-card} Job
+:link: ./job
+:link-type: doc
+
+Benchmark Job
+:::
+
+:::{grid-item-card} Generator
+:link: ./generator
+:link-type: doc
+
+Benchmark Job Generator
+:::
+
+::::
+
+```{toctree}
+:caption: Benchmark
+:hidden:
+
+./job
+./generator
+```

docs/source/markdown/guides/reference/pipelines/benchmark/job.md

@@ -0,0 +1,10 @@
+# Benchmark Job
+
+```{eval-rst}
+
+.. autoclass:: anomalib.pipelines.benchmark.job.BenchmarkJob
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+```

docs/source/markdown/guides/reference/pipelines/index.md

@@ -0,0 +1,83 @@
+# Pipelines
+
+```{danger}
+The pipelines feature is experimental and might be changed without backward compatibility.
+```
+
+## Introduction
+
+Tasks such as Benchmarking, Ensemble Tiling, and Hyper-parameter optimization requires running multiple models and chaining multiple stages together. The pipelines feature provides a way to define and run such tasks. Each part of the pipeline is designed to be independent and composable so that they can be reused across different pipelines.
+
+## Terminology
+
+- **Pipeline**: Pipeline is the main entity that defines the sequence of [jobs](./base/job.md) to be executed. It is responsible for creating and running the jobs. The job itself is generated using a [job generator](./base/generator.md). And, these are chained using a [runner](./runners/index.md).
+
+- **Runner**: A runner is responsible for scheduling and running the jobs. It also passes the output of the previous job, if available. It also calls the right hooks to gather and save the results from the jobs and passes the gathered results to the next runner.
+
+- **Job Generator**: The job generator is responsible for generating jobs based on the configuration. It is used by the runner to create jobs.
+
+- **Job**: A job is an atomic unit of work that can be run independently. It is responsible for running a single task. For example, training a model or computing metrics. The idea behind this is to ensure that it can be attached to any runner without making changes to the job itself. This is useful when you want to distribute the jobs to increase the throughput of your pipeline.
+
+```{admonition} Detailed Walkthrough
+:class: tip
+For more clarity on creating a custom pipeline, refer to the [How-To Guide](../../how_to/pipelines/index.md).
+```
+
+## Base classes
+
+::::{grid}
+:margin: 1 1 0 0
+:gutter: 1
+
+:::{grid-item-card} {octicon}`workflow` Pipeline
+:link: ./base/pipeline
+:link-type: doc
+
+Base class for pipeline.
+:::
+
+:::{grid-item-card} {octicon}`file` Job
+:link: ./base/job
+:link-type: doc
+
+Base class for job.
+:::
+
+:::{grid-item-card} {octicon}`iterations` Job Generator
+:link: ./base/generator
+:link-type: doc
+
+Base class for job generator.
+:::
+
+:::{grid-item-card} {octicon}`play` Runner
+:link: ./runners/index
+:link-type: doc
+
+Base class for runner.
+:::
+
+::::
+
+## Available Pipelines
+
+::::{grid}
+:margin: 1 1 0 0
+:gutter: 1
+
+:::{grid-item-card} {octicon}`number` Benchmarking
+:link: ./benchmark/index
+:link-type: doc
+
+Compute metrics across models using a grid-search.
+:::
+
+::::
+
+```{toctree}
+:caption: Pipelines
+:hidden:
+
+./benchmark/index
+./runners/index
+```

docs/source/markdown/guides/reference/pipelines/runners/index.md

@@ -0,0 +1,39 @@
+# Runner
+
+```{eval-rst}
+.. autoclass:: anomalib.pipelines.components.base.runner.Runner
+    :members:
+
+```
+
+## Available Runners
+
+Anomalib provides a few runners that can be used in your pipelines.
+
+::::{grid}
+:margin: 1 1 0 0
+:gutter: 1
+
+:::{grid-item-card} {octicon}`list-ordered` Serial Runner
+:link: ./serial
+:link-type: doc
+
+Runner for serial jobs.
+:::
+
+:::{grid-item-card} {octicon}`git-branch` Parallel Runner
+:link: ./parallel
+:link-type: doc
+
+Runner for parallel jobs.
+:::
+
+::::
+
+```{toctree}
+:caption: Runners
+:hidden:
+
+./serial
+./parallel
+```

docs/source/markdown/guides/reference/pipelines/runners/parallel.md

@@ -0,0 +1,7 @@
+# Parallel Runner
+
+```{eval-rst}
+.. automodule:: anomalib.pipelines.components.runners.parallel
+   :members:
+   :show-inheritance:
+```

docs/source/markdown/guides/reference/pipelines/runners/serial.md

@@ -0,0 +1,7 @@
+# Serial Runner
+
+```{eval-rst}
+.. automodule:: anomalib.pipelines.components.runners.serial
+   :members:
+   :show-inheritance:
+```

docs/source/snippets/pipelines/benchmark/cli_anomalib.txt

@@ -0,0 +1,2 @@
+# Using Anomalib entrypoint
+anomalib benchmark --config tools/experimental/benchmarking/sample.yaml

docs/source/snippets/pipelines/benchmark/cli_tools.txt

@@ -0,0 +1,2 @@
+# Using Entrypoint in tools
+python tools/experimental/benchmarking/benchmark.py --config tools/experimental/benchmarking/sample.yaml

src/anomalib/pipelines/components/base/runner.py

@@ -11,11 +11,35 @@
 
 
 class Runner(ABC):
-    """Base runner."""
+    """Base runner.
+
+    Args:
+        generator (JobGenerator): Job generator.
+    """
 
     def __init__(self, generator: JobGenerator) -> None:
         self.generator = generator
 
     @abstractmethod
     def run(self, args: dict, prev_stage_results: PREV_STAGE_RESULT = None) -> GATHERED_RESULTS:
-        """Run the pipeline."""
+        """Run the pipeline.
+
+        Args:
+            args (dict): Arguments specific to the job. For example, if there is a pipeline defined where one of the job
+                generators is hyperparameter optimization, then the pipeline configuration file will look something like
+                ```yaml
+                arg1:
+                arg2:
+                hpo:
+                    param1:
+                    param2:
+                    ...
+                ```
+                In this case, the `args` will receive a dictionary with all keys under `hpo`.
+
+            prev_stage_results (PREV_STAGE_RESULT, optional): Previous stage results. This is useful when the current
+                stage depends on the results of the previous stage. Defaults to None.
+
+        Returns:
+            GATHERED_RESULTS: Gathered results from all the jobs.
+        """