Skip to content

Commit

Permalink
Fix Readmes (#46)
Browse files Browse the repository at this point in the history 
* Add images and average

* Copy new metrics to main readme

* Added sample image to the main file

* shrunk the image

* Add caption

* Update README.md

* Intermediate PR changes

* Address issues in README

* Updated README.md

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>
  • Loading branch information
@ashwinvaidya17 @samet-akcay
3 people committed Dec 24, 2021
1 parent 0d5d26d commit 5bb4b61
Show file tree
Hide file tree
Showing 25 changed files with 375 additions and 55 deletions.
128 changes: 103 additions & 25 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ ___
[Docs](https://openvinotoolkit.github.io/anomalib) •
[License](https://github.com/openvinotoolkit/anomalib/blob/development/LICENSE)

[![python](https://img.shields.io/badge/python-3.8%2B-green)]()
[![pytorch](https://img.shields.io/badge/pytorch-1.8.1%2B-orange)]()
[![openvino](https://img.shields.io/badge/openvino-2021.4.2-purple)]()
[![black](https://img.shields.io/badge/code%20style-black-000000.svg)]()
[![python](https://img.shields.io/badge/python-3.6%2B-green)]()
[![pytorch](https://img.shields.io/badge/pytorch-1.7.1%2B-orange)]()
[![openvino](https://img.shields.io/badge/openvino-2021.4-purple)]()
[![Code Quality and Coverage](https://github.com/openvinotoolkit/anomalib/actions/workflows/tox.yml/badge.svg)](https://github.com/openvinotoolkit/anomalib/actions/workflows/tox.yml)
[![Build Docs](https://github.com/openvinotoolkit/anomalib/actions/workflows/docs.yml/badge.svg)](https://github.com/openvinotoolkit/anomalib/actions/workflows/docs.yml)

Expand All @@ -22,70 +22,148 @@ ___
___

## Introduction

Anomalib is a deep learning library that aims to collect state-of-the-art anomaly detection algorithms for benchmarking on both public and private datasets. Anomalib provides several ready-to-use implementations of anomaly detection algorithms described in the recent literature, as well as a set of tools that facilitate the development and implementation of custom models. The library has a strong focus on image-based anomaly detection, where the goal of the algorithm is to identify anomalous images, or anomalous pixel regions within images in a dataset. Anomalib is constantly updated with new algorithms and training/inference extensions, so keep checking!

##### Key features:
![Sample Image](./docs/source/images/readme.png)

**Key features:**

- The largest public collection of ready-to-use deep learning anomaly detection algorithms and benchmark datasets.
- [**PyTorch Lightning**](https://www.pytorchlightning.ai/) based model implementations to reduce boilerplate code and limit the implementation efforts to the bare essentials.
- All models can be exported to [**OpenVINO**](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) Intermediate Representation (IR) for accelerated inference on intel hardware.
- A set of [inference tools](#inference) for quick and easy deployment of the standard or custom anomaly detection models.

___

## Getting Started
The repo is thoroughly tested based on the following configuration.
* Ubuntu 20.04
* NVIDIA GeForce RTX 3090

You will need [Anaconda](https://www.anaconda.com/products/individual) installed on your system before proceeding with the Anomaly Library install.
To get an overview of all the devices where `anomalib` as been tested thoroughly, look at the [Supported Hardware](https://openvinotoolkit.github.io/anomalib/#supported-hardware) section in the documentation.

After downloading the Anomaly Library, extract the files and navigate to the extracted location.
### PyPI Install

To perform an installation, run the following:
You can get started with `anomalib` by just using pip.

```bash
pip install anomalib
```
yes | conda create -n anomalib python=3.8
conda activate anomalib
pip install -r requirements/base.txt

### Local Install
It is highly recommended to use virtual environment when installing anomalib. For instance, with [anaconda](https://www.anaconda.com/products/individual), `anomalib` could be installed as,

```bash
yes | conda create -n anomalib_env python=3.8
conda activate anomalib_env
git clone https://github.com/openvinotoolkit/anomalib.git
cd anomalib
pip install -e .
```

## Training

By default [`python tools/train.py`](https://gitlab-icv.inn.intel.com/algo_rnd_team/anomaly/-/blob/development/train.py)
runs [PADIM](https://arxiv.org/abs/2011.08785) model [MVTec](https://www.mvtec.com/company/research/datasets/mvtec-ad) `leather` dataset.
```

```bash
python tools/train.py # Train PADIM on MVTec leather
```

Training a model on a specific dataset and category requires further configuration. Each model has its own configuration
file, [`config.yaml`](https://gitlab-icv.inn.intel.com/algo_rnd_team/anomaly/-/blob/development/stfpm/anomalib/models/stfpm/config.yaml)
, which contains data, model and training configurable parameters. To train a specific model on a specific dataset and
category, the config file is to be provided:
```

```bash
python tools/train.py --model_config_path <path/to/model/config.yaml>
```

Alternatively, a model name could also be provided as an argument, where the scripts automatically finds the corresponding config file.
For example, to train [STFPM](anomalib/models/stfpm) you can use

```bash
python tools/train.py --model_config_path anomalib/models/stfpm/config.yaml
```

Alternatively, a model name could also be provided as an argument, where the scripts automatically finds the corresponding config file.

```bash
python tools/train.py --model stfpm
```

where the currently available models are:
* [DFKDE](anomalib/models/dfkde)
* [DFM](anomalib/models/dfm)
* [PADIM](anomalib/models/padim)
* [PatchCore](anomalib/models/patchcore)
* [STFPM](anomalib/models/stfpm)

- [DFKDE](anomalib/models/dfkde)
- [DFM](anomalib/models/dfm)
- [PADIM](anomalib/models/padim)
- [PatchCore](anomalib/models/patchcore)
- [STFPM](anomalib/models/stfpm)

## Inference

Anomalib contains several tools that can be used to perform inference with a trained model. The script in [`tools/inference`](tools/inference.py) contains an example of how the inference tools can be used to generate a prediction for an input image.

The following command can be used to run inference from the command line:

```bash
python tools/inference.py \
--model_config_path <path/to/model/config.yaml> \
--weight_path <path/to/weight/file> \
--image_path <path/to/image>
```
python tools/inference.py --model_config_path <path/to/model/config.yaml> --weight_path <path/to/weight/file> --image_path <path/to/image>

As a quick example:

```bash
python tools/inference.py \
--model_config_path anomalib/models/padim/config.yaml \
--weight_path results/padim/mvtec/bottle/weights/model.ckpt \
--image_path datasets/MVTec/bottle/test/broken_large/000.png
```

If the specified weight path points to a PyTorch Lightning checkpoint file (`.ckpt`), inference will run in PyTorch. If the path points to an ONNX graph (`.onnx`) or OpenVINO IR (`.bin` or `.xml`), inference will run in OpenVINO.

## Datasets
___

## Add a custom model
## Datasets

## Contributing
### [MVTec Dataset](https://www.mvtec.com/company/research/datasets/mvtec-ad)

### Image-Level AUC

| Model | | Avg | Carpet | Grid | Leather | Tile | Wood | Bottle | Cable | Capsule | Hazelnut | Metal Nut | Pill | Screw | Toothbrush | Transistor | Zipper |
| --------- | ------------------ | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :--------: | :--------: | :-------: |
| STFPM | ResNet-18 | 0.893 | 0.954 | **0.982** | 0.989 | 0.949 | 0.961 | 0.979 | 0.838 | 0.759 | **0.999** | 0.956 | 0.705 | 0.835 | **0.997** | 0.853 | 0.645 |
| STFPM | Wide ResNet-50 | 0.876 | 0.957 | 0.977 | 0.981 | 0.976 | 0.939 | 0.987 | 0.878 | 0.732 | 0.995 | 0.973 | 0.652 | 0.825 | 0.5 | 0.875 | 0.899 |
| PatchCore | ResNet-18 | 0.819 | 0.947 | 0.722 | **0.997** | 0.982 | 0.988 | 0.972 | 0.810 | 0.586 | 0.981 | 0.631 | 0.780 | 0.482 | 0.827 | 0.733 | 0.844 |
| PatchCore | Wide ResNet-50 | 0.877 | 0.981 | 0.842 | 1.0 | **0.991** | 0.991 | 0.985 | 0.868 | 0.763 | 0.988 | 0.914 | 0.769 | 0.427 | 0.806 | 0.878 | **0.958** |
| PaDiM | ResNet-18 | 0.891 | 0.945 | 0.857 | 0.982 | 0.950 | 0.976 | 0.994 | 0.844 | 0.901 | 0.750 | 0.961 | 0.863 | 0.759 | 0.889 | 0.920 | 0.780 |
| **PaDiM** | **Wide ResNet-50** | **0.950** | **0.995** | 0.942 | 1.0 | 0.974 | **0.993** | **0.999** | 0.878 | **0.927** | 0.964 | **0.989** | **0.939** | **0.845** | 0.942 | **0.976** | 0.882 |
| DFM | ResNet-18 | 0.894 | 0.864 | 0.558 | 0.945 | 0.984 | 0.946 | 0.994 | **0.913** | 0.871 | 0.979 | 0.941 | 0.838 | 0.761 | 0.95 | 0.911 | 0.949 |
| DFM | Wide ResNet-50 | 0.891 | 0.978 | 0.540 | 0.979 | 0.977 | 0.974 | 0.990 | 0.891 | 0.931 | 0.947 | 0.839 | 0.809 | 0.700 | 0.911 | 0.915 | 0.981 |
| DFKDE | ResNet-18 | 0.762 | 0.646 | 0.577 | 0.669 | 0.965 | 0.863 | 0.951 | 0.751 | 0.698 | 0.806 | 0.729 | 0.607 | 0.694 | 0.767 | 0.839 | 0.866 |
| DFKDE | Wide ResNet-50 | 0.774 | 0.708 | 0.422 | 0.905 | 0.959 | 0.903 | 0.936 | 0.746 | 0.853 | 0.736 | 0.687 | 0.749 | 0.574 | 0.697 | 0.843 | 0.892 |

### Pixel-Level AUC

| Model | | Avg | Carpet | Grid | Leather | Tile | Wood | Bottle | Cable | Capsule | Hazelnut | Metal Nut | Pill | Screw | Toothbrush | Transistor | Zipper |
| --------- | ------------------ | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :--------: | :--------: | :-------: |
| STFPM | ResNet-18 | 0.951 | 0.986 | 0.988 | 0.991 | 0.946 | 0.949 | 0.971 | 0.898 | 0.962 | 0.981 | 0.942 | 0.878 | 0.983 | 0.983 | 0.838 | 0.972 |
| STFPM | Wide ResNet-50 | 0.903 | 0.987 | **0.989** | 0.980 | **0.966** | 0.956 | 0.966 | 0.913 | 0.956 | 0.974 | 0.961 | 0.946 | **0.988** | 0.178 | 0.807 | 0.980 |
| PatchCore | ResNet-18 | 0.935 | 0.979 | 0.843 | 0.989 | 0.934 | 0.925 | 0.956 | 0.923 | 0.942 | 0.967 | 0.913 | 0.931 | 0.924 | 0.958 | 0.881 | 0.954 |
| PatchCore | Wide ResNet-50 | 0.955 | 0.988 | 0.903 | 0.990 | 0.957 | 0.936 | 0.972 | 0.950 | 0.968 | 0.974 | 0.960 | 0.948 | 0.917 | 0.969 | 0.913 | 0.976 |
| PaDiM | ResNet-18 | 0.968 | 0.984 | 0.918 | **0.994** | 0.934 | 0.947 | 0.983 | 0.965 | 0.984 | 0.978 | 0.970 | 0.957 | 0.978 | 0.988 | 0.968 | 0.979 |
| **PaDiM** | **Wide ResNet-50** | **0.979** | **0.991** | 0.970 | 0.993 | 0.955 | **0.957** | **0.985** | **0.970** | **0.988** | **0.985** | **0.982** | **0.966** | **0.988** | **0.991** | **0.976** | **0.986** |

### Image F1 Score

| Model | | Avg | Carpet | Grid | Leather | Tile | Wood | Bottle | Cable | Capsule | Hazelnut | Metal Nut | Pill | Screw | Toothbrush | Transistor | Zipper |
| --------- | ------------------ | :-------: | :-------: | :-------: | :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :--------: | :--------: | :-------: |
| STFPM | ResNet-18 | 0.932 | 0.961 | **0.982** | 0.989 | 0.930 | 0.951 | 0.984 | 0.819 | 0.918 | **0.993** | 0.973 | 0.918 | 0.887 | 0.984 | 0.790 | 0.908 |
| STFPM | Wide ResNet-50 | 0.926 | 0.973 | 0.973 | 0.974 | 0.965 | 0.929 | 0.976 | 0.853 | 0.920 | 0.972 | 0.974 | 0.922 | 0.884 | 0.833 | 0.815 | 0.931 |
| PatchCore | ResNet-18 | 0.896 | 0.933 | 0.857 | 0.995 | 0.964 | **0.983** | 0.959 | 0.790 | 0.908 | 0.964 | 0.903 | 0.916 | 0.853 | 0.866 | 0.653 | 0.898 |
| PatchCore | Wide ResNet-50 | 0.923 | 0.961 | 0.875 | **1.0** | **0.989** | 0.975 | 0.984 | 0.832 | 0.908 | 0.972 | 0.920 | 0.922 | 0.853 | 0.862 | 0.842 | 0.953 |
| PaDiM | ResNet-18 | 0.916 | 0.930 | 0.893 | 0.984 | 0.934 | 0.952 | 0.976 | 0.858 | 0.960 | 0.836 | 0.974 | 0.932 | 0.879 | 0.923 | 0.796 | 0.915 |
| **PaDiM** | **Wide ResNet-50** | **0.951** | **0.989** | 0.930 | **1.0** | 0.960 | **0.983** | **0.992** | 0.856 | **0.982** | 0.937 | **0.978** | **0.946** | **0.895** | **0.952** | **0.914** | 0.947 |
| DFM | ResNet-18 | 0.919 | 0.895 | 0.844 | 0.926 | 0.971 | 0.948 | 0.977 | **0.874** | 0.935 | 0.957 | 0.958 | 0.921 | 0.874 | 0.933 | 0.833 | 0.943 |
| DFM | Wide ResNet-50 | 0.918 | 0.960 | 0.844 | 0.990 | 0.970 | 0.959 | 0.976 | 0.848 | 0.944 | 0.913 | 0.912 | 0.919 | 0.859 | 0.893 | 0.815 | **0.961** |
| DFKDE | ResNet-18 | 0.872 | 0.864 | 0.844 | 0.854 | 0.960 | 0.898 | 0.942 | 0.793 | 0.908 | 0.827 | 0.894 | 0.916 | 0.859 | 0.853 | 0.756 | 0.916 |
| DFKDE | Wide ResNet-50 | 0.875 | 0.907 | 0.844 | 0.905 | 0.945 | 0.914 | 0.946 | 0.790 | 0.914 | 0.817 | 0.894 | 0.922 | 0.855 | 0.845 | 0.722 | 0.910 |
34 changes: 32 additions & 2 deletions anomalib/models/dfkde/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,39 @@
# Deep Feature Kernel Density Estimation

Model Type: Classification

## Description

Fast anomaly classification algorithm that consists of a deep feature extraction stage followed by anomaly classification stage consisting of PCA and Gaussian Kernel Density Estimation.

# Feature Extraction
### Feature Extraction

Features are extracted by feeding the images through a ResNet50 backbone, which was pre-trained on ImageNet. The output of the penultimate layer (average pooling layer) of the network is used to obtain a semantic feature vector with a fixed length of 2048.

# Anomaly Detection
### Anomaly Detection

In the anomaly classification stage, the features are first reduced to the first 16 principal components. Gaussian Kernel Density is then used to obtain an estimate of the probability density of new examples, based on the collection of training features obtained during the training phase.

## Usage

`python tools/train.py --model dfkde`

## Benchmark

All results gathered with seed `42`.

## [MVTec Dataset](https://www.mvtec.com/company/research/datasets/mvtec-ad)

### Image-Level AUC

| | Avg | Carpet | Grid | Leather | Tile | Wood | Bottle | Cable | Capsule | Hazelnut | Metal Nut | Pill | Screw | Toothbrush | Transistor | Zipper |
| -------------- | :---: | :----: | :---: | :-----: | :---: | :---: | :----: | :---: | :-----: | :------: | :-------: | :---: | :---: | :--------: | :--------: | :----: |
| ResNet-18 | 0.762 | 0.646 | 0.577 | 0.669 | 0.965 | 0.863 | 0.951 | 0.751 | 0.698 | 0.806 | 0.729 | 0.607 | 0.694 | 0.767 | 0.839 | 0.866 |
| Wide ResNet-50 | 0.774 | 0.708 | 0.422 | 0.905 | 0.959 | 0.903 | 0.936 | 0.746 | 0.853 | 0.736 | 0.687 | 0.749 | 0.574 | 0.697 | 0.843 | 0.892 |

### Image F1 Score

| | Avg | Carpet | Grid | Leather | Tile | Wood | Bottle | Cable | Capsule | Hazelnut | Metal Nut | Pill | Screw | Toothbrush | Transistor | Zipper |
| -------------- | :---: | :----: | :---: | :-----: | :---: | :---: | :----: | :---: | :-----: | :------: | :-------: | :---: | :---: | :--------: | :--------: | :----: |
| ResNet-18 | 0.872 | 0.864 | 0.844 | 0.854 | 0.960 | 0.898 | 0.942 | 0.793 | 0.908 | 0.827 | 0.894 | 0.916 | 0.859 | 0.853 | 0.756 | 0.916 |
| Wide ResNet-50 | 0.875 | 0.907 | 0.844 | 0.905 | 0.945 | 0.914 | 0.946 | 0.790 | 0.914 | 0.817 | 0.894 | 0.922 | 0.855 | 0.845 | 0.722 | 0.910 |
1 change: 1 addition & 0 deletions anomalib/models/dfkde/config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ dataset:

model:
name: dfkde
backbone: resnet18
max_training_points: 40000
confidence_threshold: 0.5
pre_processing: scale
Expand Down
5 changes: 3 additions & 2 deletions anomalib/models/dfkde/model.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@
from typing import Any, Dict, List, Union

import torch
import torchvision
from omegaconf.dictconfig import DictConfig
from omegaconf.listconfig import ListConfig
from torchvision.models import resnet50

from anomalib.core.model import AnomalyModule
from anomalib.core.model.feature_extractor import FeatureExtractor
Expand All @@ -34,7 +34,8 @@ def __init__(self, hparams: Union[DictConfig, ListConfig]):
self.threshold_steepness = 0.05
self.threshold_offset = 12

self.feature_extractor = FeatureExtractor(backbone=resnet50(pretrained=True), layers=["avgpool"]).eval()
self.backbone = getattr(torchvision.models, hparams.model.backbone)
self.feature_extractor = FeatureExtractor(backbone=self.backbone(pretrained=True), layers=["avgpool"]).eval()

self.normality_model = NormalityModel(
filter_count=hparams.model.max_training_points,
Expand Down
Loading

0 comments on commit 5bb4b61

Please sign in to comment.