Skip to content

cjy513203427/IADBE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

80 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ“– Introduction

It is worth noting that previous research accompanying open-source projects often takes a lot of time due to problems with environment deployment, and some projects do not work due to versioning problems. However, in this project the solution will be deployed using Docker or Conda. The primary goal is to provide researchers with a ready-to-use industrial anomaly detection platform. This platform based on anomalib should be able to reproduce and identify previous research, be bug free and easy to deploy.

πŸ“¦ Installation

Tested on Linux (Ubuntu22/20), Windows (Win11/10) βœ…

IADBE offers two ways to install the library: Conda and Docker. Use Conda if you want to make changes to dependencies and work in dev mode. Use Docker if you want to copy our environment(python, torch...) exactly. ⚠️We assume that you have installed the nvidia driver and CUDA. Otherwise, you can train on CPU.

Install from Conda Installing the library with Conda
# Use of virtual environment is highly recommended
# Using conda
conda create -n IADBE python=3.10
conda activate IADBE

# Clone the repository and install in editable mode
git clone https://github.com/cjy513203427/IADBE.git
cd IADBE

# Install anomalib
pip install anomalib

# Install the full package, this will install Anomalib CLI. Anomalib CLI is a command line interface for training, testing.
anomalib install

# Or using your favorite virtual environment
# ...
Install from Docker Installing the library with Docker
# Clone the repository and install in editable mode
git clone https://github.com/cjy513203427/IADBE.git
cd IADBE

# Build docker image
docker build --no-cache -t iadbe .
# Run docker container
docker run --gpus all -it --rm iadbe bash
You can either use it as a virtual machine with the same command to train, test and inference or set docker env as your external environment.

🧠 Training and Testing

IADBE supports both API and CLI-based training. The API is more flexible and allows for more customization, while the CLI training utilizes command line interfaces, and might be easier for those who would like to use IADBE off-the-shelf.

Training and Testing via API A train_test_mvtec_xxx.py file looks like this. Run it with your IDE or python train_test_mvtec_xxx.py to start training default with whole MVTec dataset.
import logging
from anomalib import TaskType
from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim

# configure logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

datasets = ['screw', 'pill', 'capsule', 'carpet', 'grid', 'tile', 'wood', 'zipper', 'cable', 'toothbrush', 'transistor',
            'metal_nut', 'bottle', 'hazelnut', 'leather']

for dataset in datasets:
    logger.info(f"================== Processing dataset: {dataset} ==================")
    model = Padim()
    datamodule = MVTec(category=dataset, num_workers=0, train_batch_size=256,
                       eval_batch_size=256)
    engine = Engine(pixel_metrics=["AUROC", "PRO"], image_metrics=["AUROC", "PRO"], task=TaskType.SEGMENTATION)

    logger.info(f"================== Start training for dataset: {dataset} ==================")
    engine.fit(model=model, datamodule=datamodule)

    logger.info(f"================== Start testing for dataset: {dataset} ==================")
    test_results = engine.test(
        model=model,
        datamodule=datamodule,
        ckpt_path=engine.trainer.checkpoint_callback.best_model_path,
    )
Training and Testing via CLI A train_test_mvtec_xxx.sh file looks like this. Run it with bash train_test_mvtec_xxx.sh to start training default with whole MVTec dataset.
#!/bin/bash

datasets=('screw' 'pill' 'capsule' 'carpet' 'grid' 'tile' 'wood' 'zipper' 'cable' 'toothbrush' 'transistor' 'metal_nut' 'bottle' 'hazelnut' 'leather')
config_file="./configs/models/padim.yaml"

for dataset in "${datasets[@]}"
do
    command="anomalib train --data anomalib.data.MVTec --data.category $dataset --config $config_file"
    echo "Running command: $command"
    # Excute command
    $command
done

For the futher use of anomalib cli, you can retrieve Training via CLI from Training

πŸ€– Inference

Anomalib includes multiple inferencing scripts, including Torch, Lightning, Gradio, and OpenVINO inferencers to perform inference using the trained/exported model. Here we show an inference example using the Lightning inferencer.

Inference via API

The following example demonstrates how to perform Lightning inference by loading a model from a checkpoint file.

# Assuming the datamodule, custom_model and engine is initialized from the previous step,
# a prediction via a checkpoint file can be performed as follows:
predictions = engine.predict(
    datamodule=datamodule,
    model=model,
    ckpt_path="path/to/checkpoint.ckpt",
)
Inference via CLI
# To get help about the arguments, run:
anomalib predict -h

# Predict by using the default values.
anomalib predict --custom_model anomalib.models.Patchcore \
                 --data anomalib.data.MVTec \
                 --ckpt_path <path/to/custom_model.ckpt>

# Predict by overriding arguments.
anomalib predict --custom_model anomalib.models.Patchcore \
                 --data anomalib.data.MVTec \
                 --ckpt_path <path/to/custom_model.ckpt>
                 --return_predictions

# Predict by using a config file.
anomalib predict --config <path/to/config> --return_predictions

πŸ“œ Custom Dataset

IADBE can help you with training and inference for your own custom dataset.

  1. The first thing you need to do is import your own dataset to the project and create a custom data configuration file.
Configuration with only Normal Images
class_path: anomalib.data.Folder
init_args:
  name: "custom_dataset"
  root: "datasets/Custom_Dataset/hazelnut"
  normal_dir: "train/good"
  abnormal_dir: "test/crack"
  mask_dir: null
  normal_split_ratio: 0.2
  test_split_mode: synthetic
Configuration with Normal and Abnormal Images
class_path: anomalib.data.Folder
init_args:
  name: "custom_dataset"
  root: "datasets/Custom_Dataset/chest_xray"
  normal_dir: "train/good"
  abnormal_dir: "test/crack"
  normal_test_dir: "test/good"
  normal_split_ratio: 0
  extensions: [".png"]
  image_size: [256, 256]
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  task: classification
  train_transform: null
  eval_transform: null
  test_split_mode: synthetic
  test_split_ratio: 0.2
  val_split_mode: same_as_test
  val_split_ratio: 0.5
  seed: null
  1. Next, you'll want to choose a model and then train it with a custom dataset.
anomalib train --data <path/to/custom_dataset.yaml> --model anomalib.models.Padim 
  1. Finally, you can run an inference with the trained model to get the predicted results.
anomalib predict --model anomalib.models.Padim \
                 --data <path/to/custom_dataset/custom_image.png> \
                 --ckpt_path <path/to/custom_dataset.ckpt>

πŸ“Š Benchmark Results

These are partial results of benchmark. You can find more details in my paper, which I'll upload later. The results come from raw logs, which you can find under logs/rawlogs.

Image Level AUROC on MVTec

Screw Pill Capsule Carpet Grid Tile Wood Zipper Cable Toothbrush Transistor Metal Nut Bottle Hazelnut Leather Average
CFA 87.68 93.10 48.17 50.00 84.04 37.50 62.50 40.34 61.96 98.06 45.00 39.78 50.00 60.00 50.00 60.54
CFLOW 81.57 92.91 95.45 97.27 88.47 100.00 99.39 97.64 94.15 94.17 94.88 100.00 100.00 99.68 100.00 95.71
CSFLOW 41.58 38.09 60.77 99.16 78.36 90.44 95.75 88.93 67.31 44.72 50.54 74.46 89.92 71.02 100.00 72.74
DFKDE 69.17 64.18 72.04 68.42 46.69 92.50 81.32 88.26 68.67 78.89 81.12 76.20 92.94 77.57 77.79 75.72
DFM 77.79 96.89 92.98 90.93 65.41 98.67 97.81 97.51 94.27 96.39 94.79 91.72 100.00 96.82 100.00 92.80
DRAEM 30.01 74.55 77.02 72.47 80.45 86.80 95.96 78.90 63.98 70.42 90.21 93.55 98.10 77.50 86.65 78.44
DSR 56.67 70.16 72.72 43.82 97.08 80.70 90.75 78.97 76.96 96.94 91.04 81.04 86.59 81.64 83.97 79.27
FASTFLOW 65.85 76.60 69.64 93.38 96.32 93.36 98.16 72.69 67.62 72.50 89.62 81.33 99.68 79.86 99.90 83.77
GANomaly 32.79 59.96 26.57 21.71 57.23 54.42 60.88 41.05 52.47 49.17 33.46 26.30 47.78 53.86 36.68 43.62
PaDiM 78.95 79.95 86.48 97.99 87.47 94.55 97.46 77.46 85.96 82.50 94.54 98.34 99.52 88.32 100.00 89.97
PatchCore 98.11 94.76 97.85 99.12 98.08 98.81 98.77 99.21 99.10 100.00 100.00 99.80 100.00 100.00 100.00 98.91
RD 98.03 97.63 97.93 99.36 95.49 100.00 99.39 97.16 95.45 91.39 97.87 100.00 100.00 100.00 100.00 97.98
RKDE 50.58 68.77 51.93 - 75.36 67.72 62.54 75.37 85.83 77.17 65.00 90.63 85.07 100.00 100.00 75.43
STFPM 77.62 40.78 60.59 98.88 59.23 97.08 98.95 75.39 91.34 47.50 61.88 40.22 43.97 96.50 100.00 72.66
UFLOW 49.19 94.84 56.72 100.00 99.33 99.39 95.09 89.73 62.67 64.17 81.46 55.77 99.21 90.39 100.00 82.53

Image Level AUROC on VisA

Candle Capsules Cashew Chewinggum Fryum Macaroni1 Macaroni2 PCB1 PCB2 PCB3 PCB4 Pipe Fryum Average
CFA 85.49 12.67 19.12 96.12 19.12 88.20 59.00 94.91 93.31 90.27 99.04 19.12 75.32
CFLOW 89.24 94.22 95.60 99.66 80.48 74.59 66.63 87.86 88.12 74.47 96.50 98.12 87.12
CSFLOW 36.69 55.96 41.39 67.73 39.54 52.00 38.75 63.75 49.53 55.11 75.28 31.83 52.46
DFKDE 85.91 59.15 86.32 82.86 75.94 65.47 45.47 67.19 66.20 69.61 83.19 77.79 72.09
DFM 95.23 57.63 95.86 98.46 95.22 76.92 70.17 93.07 86.92 85.45 97.47 90.04 87.21
DRAEM 77.33 73.80 89.16 72.54 79.02 77.44 66.49 72.96 90.77 88.99 95.67 82.06 80.52
DSR 65.80 77.63 88.80 91.94 83.54 81.14 65.81 95.65 96.31 94.81 98.29 78.35 84.84
FASTFLOW 95.24 78.87 89.00 98.28 92.82 88.41 75.74 88.93 88.10 85.57 96.34 92.00 89.11
GANomaly 69.67 67.23 80.96 56.78 89.52 64.24 49.48 27.38 31.19 22.16 54.83 49.58 55.25
PaDiM 81.06 64.85 91.30 97.66 87.90 83.42 69.71 86.34 84.92 75.91 95.91 91.28 84.19
PatchCore 98.34 72.48 96.32 99.22 95.60 86.97 71.52 94.20 93.86 93.11 99.07 99.24 91.66
RD 94.22 87.52 96.90 99.42 90.62 96.21 84.77 95.79 97.02 96.51 99.89 98.46 94.78
RKDE 73.97 60.90 85.16 64.46 77.98 62.51 46.59 68.51 78.08 74.20 49.55 73.43 67.95
STFPM 76.44 88.17 59.22 91.32 93.80 74.04 89.62 95.22 73.44 90.73 92.13 95.30 84.95
UFLOW 92.87 87.45 94.94 99.34 93.62 97.80 78.58 95.85 96.27 96.83 96.90 97.46 93.99

Image Level AUROC on MVTec3D

Bagel Cable Gland Carrot Cookie Dowel Foam Peach Potato Rope Tire Average
CFA 8.52 53.97 19.61 7.91 46.15 86.94 78.81 56.77 92.07 0.00 45.08
CFLOW 91.43 69.35 73.48 67.06 98.82 68.56 70.39 55.73 94.79 74.16 76.38
CSFLOW 53.33 61.03 43.41 48.16 38.31 70.69 56.62 33.45 72.55 63.84 54.14
DFKDE 64.36 71.67 73.25 49.10 72.97 53.87 58.24 52.57 83.11 42.30 62.14
DFM 92.51 66.78 45.38 31.52 45.23 33.19 59.51 65.81 84.83 59.45 58.42
DRAEM 88.02 71.10 56.78 50.87 44.79 14.03 71.12 70.65 59.78 62.16 58.93
DSR 81.87 76.25 71.21 36.51 70.58 53.37 60.14 55.26 48.19 57.29 61.07
FASTFLOW 83.99 66.28 78.09 71.67 67.68 72.62 48.40 40.51 88.27 67.31 68.48
GANomaly 36.78 65.96 50.22 2.84 62.72 47.87 59.00 56.87 40.49 44.83 46.76
PaDiM 95.71 73.56 64.83 62.48 89.68 64.50 72.50 53.51 76.36 80.97 73.41
PatchCore 92.72 91.95 86.84 74.48 97.86 72.56 84.94 60.13 94.66 79.95 83.61
RD 96.18 90.75 91.44 66.40 99.33 83.56 84.43 68.38 - - 95.06
RKDE 51.81 71.43 47.00 60.85 56.32 53.87 53.92 41.90 35.85 56.28 52.92
STFPM 91.74 84.35 64.20 49.38 82.66 73.44 68.65 52.52 93.98 51.13 71.21
UFLOW 95.40 88.40 84.34 51.28 96.89 62.44 60.49 57.02 96.24 60.92 75.34

Image Level AUROC on Btech

01 02 03 Average
CFA 21.77 76.90 99.13 65.93
CFLOW 97.08 85.40 99.70 94.06
CSFLOW 25.12 73.10 29.38 42.53
DFKDE 93.88 78.40 99.14 90.47
DFM 99.32 86.57 99.99 95.29
DRAEM 91.45 80.42 85.66 85.84
DSR 95.63 73.75 95.33 88.24
FASTFLOW 99.71 85.47 90.91 92.03
GANomaly 73.18 35.00 41.97 50.05
PaDiM 99.71 87.27 99.57 95.52
PatchCore 97.76 83.03 99.99 93.59
RD 99.61 86.25 99.98 95.28
RKDE 92.03 73.78 85.61 83.81
STFPM 95.43 81.37 99.89 92.23
UFLOW 98.93 85.55 96.42 93.63

Image Level AUROC on Kolektor

CFA CFLOW CSFLOW DFKDE DFM DRAEM DSR FASTFLOW GANomaly PaDiM PatchCore RD RKDE STFPM UFLOW
Kolektor 52.77 86.65 58.70 67.84 - - 57.72 77.36 56.68 80.25 86.76 88.24 64.08 66.24 90.91

About

Industrial Anomaly Detection Benchmark Engine

Resources

Stars

Watchers

Forks

Packages

No packages published