KeyError: 'anomaly_maps'

[igsing]

Training fails with "KeyError: 'anomaly_maps'" for all the classification models (DFM, DFKDE, GANomaly). The failure is during testing stage, e.g.:

python tools/train.py --model dfm

...
Testing DataLoader 0: 0%|
...
File "/home/zingman/code/python_local/anomalib/anomalib/utils/callbacks/visualizer_callback.py", line 132, in on_test_batch_end
outputs["anomaly_maps"],
KeyError: 'anomaly_maps'
Testing DataLoader 0: 0%| | 0/3 [00:00<?, ?it/s]

The repository was cloned on 13 June.

[samet-akcay]

Hi @igsing, thanks for reporting this. We'll investigate this and give you an update here

[samet-akcay]

@igsing, this was a bug due to a misconfiguration in the config files. I've proposed a fix in this PR.

[abhinand4as]

I am getting the same error when I try to train a custom model.

Error Message:

2022-07-11 11:24:07,585 - anomalib.utils.loggers - INFO - Loading the experiment logger(s)
2022-07-11 11:24:07,585 - anomalib.utils.callbacks - INFO - Loading the callbacks
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True, used: True
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - IPU available: False, using: 0 IPUs
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - HPU available: False, using: 0 HPUs
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - Trainer(limit_train_batches=1.0) was configured so 100% of the batches per epoch will be used..
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - Trainer(limit_val_batches=1.0) was configured so 100% of the batches will be used..
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - Trainer(limit_test_batches=1.0) was configured so 100% of the batches will be used..
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - Trainer(limit_predict_batches=1.0) was configured so 100% of the batches will be used..
2022-07-11 11:24:07,622 - pytorch_lightning.utilities.rank_zero - INFO - Trainer(val_check_interval=1.0) was configured so validation will run at the end of the training epoch..
2022-07-11 11:24:07,622 - anomalib - INFO - Training the model.
2022-07-11 11:24:07,626 - anomalib.data.folder - INFO - Setting up train, validation, test and prediction datasets.
/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric ROC will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
warnings.warn(*args, **kwargs)
2022-07-11 11:24:09,484 - pytorch_lightning.accelerators.gpu - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py:183: UserWarning: LightningModule.configure_optimizers returned None, this fit will run with no optimizer
rank_zero_warn(
2022-07-11 11:24:09,486 - pytorch_lightning.callbacks.model_summary - INFO -
| Name | Type | Params

0 | image_threshold | AdaptiveThreshold | 0
1 | pixel_threshold | AdaptiveThreshold | 0
2 | training_distribution | AnomalyScoreDistribution | 0
3 | min_max | MinMax | 0
4 | model | DfkdeModel | 11.7 M
5 | image_metrics | AnomalibMetricCollection | 0
6 | pixel_metrics | AnomalibMetricCollection | 0

11.7 M Trainable params
0 Non-trainable params
11.7 M Total params
46.758 Total estimated model params size (MB)
Epoch 0: 0%| | 0/159 [00:00<?, ?it/s]/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py:137: UserWarning: training_step returned None. If this was on purpose, ignore this warning...
self.warning_cache.warn("training_step returned None. If this was on purpose, ignore this warning...")
Epoch 0: 31%|██████▉ 2022-07-11 11:24:09,912 - anomalib.models.dfkde.lightning_model - INFO - Fitting a KDE model to the embedding collected from the training set.
Epoch 0: 100%|█| 159/159 [00:01<00:00, 119.76it/s, loss=nan, image_F1Score=0.926
2022-07-11 11:24:10,924 - anomalib.utils.callbacks.timer - INFO - Training took 1.44 seconds
2022-07-11 11:24:10,924 - anomalib - INFO - Loading the best model weights.
2022-07-11 11:24:10,924 - anomalib - INFO - Testing the model.
2022-07-11 11:24:10,927 - anomalib.data.folder - INFO - Setting up train, validation, test and prediction datasets.
/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric ROC will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.
warnings.warn(*args, **kwargs)
2022-07-11 11:24:10,949 - pytorch_lightning.accelerators.gpu - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
2022-07-11 11:24:10,950 - anomalib.utils.callbacks.model_loader - INFO - Loading the model from /home/cvision/ws/ws_gadgeon/anomalib/results/dfkde/test/weights/model.ckpt
Testing DataLoader 0: 0%| | 0/109 [00:00<?, ?it/s]Traceback (most recent call last):
File "tools/train.py", line 82, in
train()
File "tools/train.py", line 78, in train
trainer.test(model=model, datamodule=datamodule)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 938, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1320, in _run_stage
return self._run_evaluate()
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1365, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 134, in advance
self._on_evaluation_batch_end(output, **kwargs)
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 267, in _on_evaluation_batch_end
self.trainer._call_callback_hooks(hook_name, output, *kwargs.values())
File "/home/cvision/.virtualenvs/anomalib_env2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1636, in _call_callback_hooks
fn(self, self.lightning_module, *args, **kwargs)
File "/home/cvision/ws/ws_gadgeon/anomalib/anomalib/utils/callbacks/visualizer_callback.py", line 149, in on_test_batch_end
for i, image in enumerate(self.visualizer.visualize_batch(outputs)):
File "/home/cvision/ws/ws_gadgeon/anomalib/anomalib/post_processing/visualizer.py", line 89, in visualize_batch
anomaly_map=batch["anomaly_maps"][i].cpu().numpy(),
KeyError: 'anomaly_maps'

Config:

dataset:
name: test #options: [mvtec, btech, folder]
format: folder
path: ./dataset
task: classification
normal_dir: normal # name of the folder containing normal images.
abnormal_dir: abnormal # name of the folder containing abnormal images.
normal_test_dir: test # name of the folder containing normal test images.
split_ratio: 0.2
mask: null
extensions: null
image_size: 128
train_batch_size: 4
test_batch_size: 1
num_workers: 8
transform_config:
train: null
val: null
create_validation_set: false

model:
name: dfkde
backbone: resnet18
max_training_points: 40000
confidence_threshold: 0.5
pre_processing: scale
n_components: 16
threshold_steepness: 0.05
threshold_offset: 12
normalization_method: min_max # options: [null, min_max, cdf]

metrics:
image:
- F1Score
- AUROC
threshold:
image_default: 0
adaptive: true

visualization:
show_images: False # show images on the screen
save_images: True # save images to the file system
log_images: True # log images to the available loggers (if any)
image_save_path: null # path to which images will be saved
mode: full # options: ["full", "simple"]

project:
seed: 42
path: ./results

logging:
logger: [] # options: [tensorboard, wandb, csv] or combinations.
log_graph: false # Logs the model graph to respective logger.

PL Trainer Args. Don't add extra parameter here.

trainer:
accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
accumulate_grad_batches: 1
amp_backend: native
auto_lr_find: false
auto_scale_batch_size: false
auto_select_gpus: false
benchmark: false
check_val_every_n_epoch: 1 # Don't validate before extracting features.
default_root_dir: null
detect_anomaly: false
deterministic: false
devices: 1
enable_checkpointing: true
enable_model_summary: true
enable_progress_bar: true
fast_dev_run: false
gpus: null # Set automatically
gradient_clip_val: 0
ipus: null
limit_predict_batches: 1.0
limit_test_batches: 1.0
limit_train_batches: 1.0
limit_val_batches: 1.0
log_every_n_steps: 50
log_gpu_memory: null
max_epochs: 1
max_steps: -1
max_time: null
min_epochs: null
min_steps: null
move_metrics_to_cpu: false
multiple_trainloader_mode: max_size_cycle
num_nodes: 1
num_processes: null
num_sanity_val_steps: 0
overfit_batches: 0.0
plugins: null
precision: 32
profiler: null
reload_dataloaders_every_n_epochs: 0
replace_sampler_ddp: true
strategy: null
sync_batchnorm: false
tpu_cores: null
track_grad_norm: -1
val_check_interval: 1.0 # Don't validate before extracting features.