-
Notifications
You must be signed in to change notification settings - Fork 646
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PatchCore results are much worth than reported #715
Comments
@spoorgholi74, looks like some of the recent changes made in patchcore score computation caused this issue. --- main
image_AUROC 0.2395666092634201
image_F1Score 0.8640776872634888
pixel_AUROC 0.9860117435455322
pixel_F1Score 0.5501166582107544
--- v0.3.7
image_AUROC 0.959470272064209
image_F1Score 0.9392265677452087
pixel_AUROC 0.986011803150177
pixel_F1Score 0.5501289963722229
--- v0.3.6
image_AUROC 0.9867576360702515
image_F1Score 0.9828571677207947
pixel_AUROC 0.9879260063171387
pixel_F1Score 0.5555583834648132
--- v0.3.5
image_AUROC 0.9815409779548645
image_F1Score 0.9714285731315613
pixel_AUROC 0.9881787300109863
pixel_F1Score 0.5748549103736877 You could use v0.3.6 until we fix the problem. |
@code10086web and @DeepSpace98, can you confirm if you are able to get higher performance when you checkout v0.3.6? |
Yes,v0.3.6 is great |
I found one bug if using a test batch size > 1. (Note that this error should already be in v0.3.7 but back then the default batch size was still 1 whereas it was changed to 32 recently, thus the huge drop.) Now I get an image-AUROC of 0.959 for "carpet", which is still worse than in the paper (~0.987) and "grid" has an image-AUROC of only 0.718, so there should be another bug somewhere. |
Can we please keep this issue open? Because as I mentioned above, the my results are better but still worse than in the paper |
hey @Jonas1302 can refer to this fix b66c0a7 |
I'm sorry, what exactly do you mean? |
i see, thought it is different people |
We managed to narrow down the performance deterioration to a small change in the Average Pooling layer that was made some time ago, and we've reverted that commit for now. I ran a quick experiment on a few MVTec categories where I compared our numbers to those obtained by running the original implementation. These are some results:
(The seed was set to 0 in both the implementations, and all other parameters were kept at default, in case anyone would like to repeat these numbers). There is still a small difference in the
So I believe that our PatchCore model is now on par with the original implementation :) |
Thank you very much! I can confirm that the results are definitely better! Just in case anyone is interested, I did a few experiments during the last few days which I wanted to share. But feel free to skip the rest of this comment. I trained different settings twice over all categories with the seeds The settings are as followed:
|
hi @Jonas1302 nice sharing, btw what do you mean by it has two errors in original embedding? so using original generate_embedding solve the problem too? and what is patchcore 10% means? is it 10% dataset? |
Thanks for sharing @Jonas1302 ! It is true that some of the results in our README are outdated. We are planning to revisit the README results for all models in the near future to make sure everything is up to date. |
I think the 10% means the embedding downsampling ratio to memory_bank,it's the "coreset_sampling_ratio" in the config. |
Describe the bug
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The image AUCROC to be .98 for the category carpet of the MvTech dataset
but it is very low.
Fastflow work as expected so the problem seems to be patchcore.
Screenshots
Hardware and Software Configuration
Log
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
----------------------------------/anomalib/config/config.py:166: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.
warn(
2022-11-16 11:52:49,662 - anomalib.data - INFO - Loading the datamodule
2022-11-16 11:52:49,662 - anomalib.pre_processing.pre_process - WARNING - Transform configs has not been provided. Images will be normalized using ImageNet statistics.
2022-11-16 11:52:49,663 - anomalib.pre_processing.pre_process - WARNING - Transform configs has not been provided. Images will be normalized using ImageNet statistics.
2022-11-16 11:52:49,663 - anomalib.models - INFO - Loading the model.
2022-11-16 11:52:49,667 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmpk5fh8j6r
2022-11-16 11:52:49,667 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmpk5fh8j6r/_remote_module_non_scriptable.py
2022-11-16 11:52:49,674 - anomalib.models.components.base.anomaly_module - INFO - Initializing PatchcoreLightning model.
/home/-/code/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric
PrecisionRecallCurve
will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.warnings.warn(*args, **kwargs)
2022-11-16 11:52:50,882 - timm.models.helpers - INFO - Loading pretrained weights from url (https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/wide_resnet50_racm-8234f177.pth)
2022-11-16 11:52:51,009 - anomalib.utils.loggers - INFO - Loading the experiment logger(s)
2022-11-16 11:52:51,009 - anomalib.utils.callbacks - INFO - Loading the callbacks
/home/-/code/anomalib/src/anomalib/anomalib/utils/callbacks/init.py:141: UserWarning: Export option: None not found. Defaulting to no model export
warnings.warn(f"Export option: {config.optimization.export_mode} not found. Defaulting to no model export")
2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True, used: True
2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores
2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO - IPU available: False, using: 0 IPUs
2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO - HPU available: False, using: 0 HPUs
2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO -
Trainer(limit_train_batches=1.0)
was configured so 100% of the batches per epoch will be used..2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO -
Trainer(limit_val_batches=1.0)
was configured so 100% of the batches will be used..2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO -
Trainer(limit_test_batches=1.0)
was configured so 100% of the batches will be used..2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO -
Trainer(limit_predict_batches=1.0)
was configured so 100% of the batches will be used..2022-11-16 11:52:51,012 - pytorch_lightning.utilities.rank_zero - INFO -
Trainer(val_check_interval=1.0)
was configured so validation will run at the end of the training epoch..2022-11-16 11:52:51,012 - anomalib - INFO - Training the model.
2022-11-16 11:52:51,016 - anomalib.data.mvtec - INFO - Found the dataset.
2022-11-16 11:52:51,018 - anomalib.data.mvtec - INFO - Setting up train, validation, test and prediction datasets.
/-/-/code/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric
ROC
will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.warnings.warn(*args, **kwargs)
2022-11-16 11:52:52,479 - pytorch_lightning.accelerators.gpu - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
/-/-/code/anomalib/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py:183: UserWarning:
LightningModule.configure_optimizers
returnedNone
, this fit will run with no optimizerrank_zero_warn(
2022-11-16 11:52:52,482 - pytorch_lightning.callbacks.model_summary - INFO -
| Name | Type | Params
0 | image_threshold | AnomalyScoreThreshold | 0
1 | pixel_threshold | AnomalyScoreThreshold | 0
2 | model | PatchcoreModel | 24.9 M
3 | image_metrics | AnomalibMetricCollection | 0
4 | pixel_metrics | AnomalibMetricCollection | 0
5 | normalization_metrics | MinMax | 0
24.9 M Trainable params
0 Non-trainable params
24.9 M Total params
99.450 Total estimated model params size (MB)
Epoch 0: 8%|▊ | 1/13 [00:01<00:16, 1.37s/it, loss=nan]/-/-/code/anomalib/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py:137: UserWarning:
training_step
returnedNone
. If this was on purpose, ignore this warning...self.warning_cache.warn("
training_step
returnedNone
. If this was on purpose, ignore this warning...")Epoch 0: 69%|██████▉ | 9/13 [00:01<00:00, 4.67it/s, loss=nan]
Validation: 0it [00:00, ?it/s]2022-11-16 11:52:54,414 - anomalib.models.patchcore.lightning_model - INFO - Aggregating the embedding extracted from the training set.
2022-11-16 11:52:54,415 - anomalib.models.patchcore.lightning_model - INFO - Applying core-set subsampling to get the embedding.
Epoch 0: 69%|██████▉ | 9/13 [00:20<00:08, 2.22s/it, loss=nan]
Validation: 0%| | 0/4 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/4 [00:00<?, ?it/s]
Validation DataLoader 0: 25%|██▌ | 1/4 [00:00<00:00, 4.13it/s]
Epoch 0: 77%|███████▋ | 10/13 [00:59<00:17, 5.94s/it, loss=nan]
Validation DataLoader 0: 50%|█████ | 2/4 [00:00<00:00, 4.04it/s]
Epoch 0: 85%|████████▍ | 11/13 [00:59<00:10, 5.42s/it, loss=nan]
Validation DataLoader 0: 75%|███████▌ | 3/4 [00:00<00:00, 4.02it/s]
Epoch 0: 92%|█████████▏| 12/13 [00:59<00:04, 4.99s/it, loss=nan]
Validation DataLoader 0: 100%|██████████| 4/4 [00:00<00:00, 4.63it/s]
Epoch 0: 100%|██████████| 13/13 [01:00<00:00, 4.67s/it, loss=nan, pixel_F1Score=0.548, pixel_AUROC=0.986]
Epoch 0: 100%|██████████| 13/13 [01:01<00:00, 4.69s/it, loss=nan, pixel_F1Score=0.548, pixel_AUROC=0.986]
2022-11-16 11:53:53,628 - anomalib.utils.callbacks.timer - INFO - Training took 61.15 seconds
2022-11-16 11:53:53,628 - anomalib - INFO - Loading the best model weights.
2022-11-16 11:53:53,628 - anomalib - INFO - Testing the model.
2022-11-16 11:53:53,632 - anomalib.data.mvtec - INFO - Found the dataset.
2022-11-16 11:53:53,633 - anomalib.data.mvtec - INFO - Setting up train, validation, test and prediction datasets.
/-/code/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric
ROC
will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.warnings.warn(*args, **kwargs)
2022-11-16 11:53:53,716 - pytorch_lightning.accelerators.gpu - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
2022-11-16 11:53:53,718 - anomalib.utils.callbacks.model_loader - INFO - Loading the model from /home/-/code/anomalib/src/anomalib/results/patchcore/mvtec/carpet/run/weights/model.ckpt
Testing DataLoader 0: 100%|██████████| 4/4 [00:19<00:00, 4.65s/it]2022-11-16 11:54:14,762 - anomalib.utils.callbacks.timer - INFO - Testing took 20.9255051612854 seconds
Throughput (batch_size=32) : 5.591262867883519 FPS
Testing DataLoader 0: 100%|██████████| 4/4 [00:19<00:00, 4.97s/it]
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Test metric DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
image_AUROC 0.4036917984485626
image_F1Score 0.8640776872634888
pixel_AUROC 0.9860672950744629
pixel_F1Score 0.5481611490249634
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Process finished with exit code 0
The text was updated successfully, but these errors were encountered: