AUC is lower than PADIM

[XiaoJiNu]

As following, auc of some classes is much less than PADIM. I think better AUC is more important than better PRO.

[Pangoraw]

It seems that https://github.com/xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master is using a center crop on the images. Do you know if this is common practice on the MVTec AD dataset ?

If you add it to the example mvtec file you get similar image ROCAUC scores (see a1b80ad for an example).

[XiaoJiNu]

@Pangoraw Sorry, I didn't know the difference. So, you meant that https://github.com/xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master uses transforms.CenterCrop(224) but you didn't, that's why your ROCAUC is lower ? On the other hand, I think it's not good to crop the center of images because anomaly is not always in the center.

[Pangoraw]

Also which backbone and number of dimensions (k) are you using for both PaDiM and this method ?

On the other hand, I think it's not good to crop the center of images because anomaly is not always in the center.

The main goal is still to compare the methods in the same setting to assess the performance. You can chose to remove or add transformations for your own application. Also note that the pixel ROCAUC (which favors big anomalies) you have on the left column is different from the PRO score metric (see this paper for more details).

[XiaoJiNu]

@Pangoraw Thanks for your reply. Another question is that the paper uses semiOrthogonal to accelerate computing speed, but I didn't find the speed comparasion. How much it is faster than not use semiOrthogonal ?

[Pangoraw]

Well, compared to methods learnt using gradient descent like a VAE or AnoGAN it is faster because you only need a single forward pass. But it will be marginally slower than PaDiM I guess.

[XiaoJiNu]

@Pangoraw Ok, I will test it more. Your job is so great !

[XiaoJiNu]

@Pangoraw Hello, I have modified the code and the result is as following picture, average auc is higher than before adding center crop but still lower than PADIM. What should I do to improve auc ? Whether auc of Semiorthogonal is indeed lower than PADIM ?

[Pangoraw]

hmm interesting.. I was able to get a better ROC AUC by computing the image score after the gaussian blur but it is still not as high as PaDiM. However, i found that changing the random seed in the PaDiM implementation yields totally different results (see xiahaifeng1995/PaDiM-Anomaly-Detection-Localization-master#15). I think this particular seed selects relevant dimensions for discriminating the MVTec dataset but may not be as good for other problems where you would need to search for a new seed whereas with the semi-orthogonal you should have a more consistant performance because all dimensions are blended.

[XiaoJiNu]

@Pangoraw Maybe seed selets is the keypoint, I will test it on other datasets in the future and discuss the result with you. Thank you !