Generalize PatchCore to support feature map sizes other than (28, 28).

[Lucas-Steinmann]

Description

This PR generalizes the PatchCore model to support different input sizes and different layers of the ResNet models.
This is implemented by inferring the pooling/downsampling ratio from the layer names.
The ratio is then used to calculate the feature map size.
The largest feature map size is used in the AnomalyMapGenerator to recreate the shape of the 2D-shape from the 1D-embedding.

It assumes the layer names and downsampling ratios are the same as in the torchvision implementation of ResNet.

• Fixes issue PatchCore only support 224x224 input, and 28x28 feature maps. #40 for ResNets.

Changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes

[samet-akcay]

@Lucas-Steinmann, thanks a ton for addressing this long-awaited issue! I only have a really minor comment!

[djdameln]

Thanks a lot for your contribution. Your solution would work, but I feel this could also be achieved without pre-computing the feature map size. Instead we could maybe infer the feature map size from the shape of the embeddings that are computed in the forward method, here, and pass it to the anomaly map generator along with the patch scores. This would leave us with a more generic solution that would work for other backbones as well.

[Lucas-Steinmann]

Thanks a lot for your contribution. Your solution would work, but I feel this could also be achieved without pre-computing the feature map size. Instead we could maybe infer the feature map size from the shape of the embeddings that are computed in the forward method, here, and pass it to the anomaly map generator along with the patch scores. This would leave us with a more generic solution that would work for other backbones as well.

Yes, that would be definitely another, more generic solution. I've also mentioned it in issue #40.
But it requires at least one call to forward.
I'm not sure, if it is ok to do this during __init__ with a "blank image". If not, the AnomalyMapGenerator maybe has to be created later during the inference on demand (and cached).

[djdameln]

@Lucas-Steinmann thanks for clarifying. I wasn't aware of the discussion in #40. Your solution is a nice improvement already, so I would not mind merging it and extending it later.

On the long term, I still think it would be better to achieve this without pre-computing the feature map size. It won't be necessary to pass a blank image or instantiate the anomaly map generator during inference. Since self.feature_map_size is only used in compute_anomaly_map, it doesn't need to be an attribute. Instead, we could pass it along with the patch scores when we call the anomaly map generator from the model's forward method.

[Lucas-Steinmann]

Oh sure, right. Good catch!

[samet-akcay]

@Lucas-Steinmann, following this PR to address the feature dimension issue in PatchCode, another community PR was created in #148. Based on the discussions here, the proposed solution in #148 is not limited to ResNet, but generic to other backbones as well. Does the solution in #148 suit your needs? If so we will close this PR and merge #148? We really appreciate your effort, kindly welcome you to keep contributing this repo.

[Lucas-Steinmann]

Yes, that's perfect.