Add ensembling methods for tiling to Anomalib #1226

blaz-r · 2023-08-01T17:21:40Z

Description

This PR adds mechanism to train ensemble of models on tiled images. It is part of Google Summer of Code.

A lot of details on implementation as well as discussion can be accessed in #1131.

Closes #1727

Changes

Bug fix (non-breaking change which fixes an issue)
Refactor (non-breaking change which refactors the code base)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

Some things still todo (tests, docs...) but most of the code is ready for review.

My code follows the pre-commit style and check guidelines of this project.
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing tests pass locally with my changes
I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

ashwinvaidya17

Thanks! Massive efforts here. I have a few minor comments regarding refactoring the config file. It is more of a personal preference. Other than that, I think there are some files that might be left over from the previous version.

ashwinvaidya17 · 2023-09-11T12:33:58Z

tools/tiled_ensemble/post_processing/postprocess.py

+        out = {}
+        for step in tqdm(self.steps):
+            if step.final_compute:
+                out[step.name] = step.compute()


From what I understand, compute is only defined for EnsembleMetrics, and is the only one for which final_compute is set to True. I have two ideas here but they are more of a personal preference,

Remove final_compute and implement compute for all sub-classes with pass or

Use is_overridden from lightning_utilities.core.overrides to explicitly check if the step has compute defined
something like
if is_overridden("compute", step): step.compute()

This is no longer used, I'll delete these files asap.

ashwinvaidya17 · 2024-06-26T08:21:38Z

src/anomalib/pipelines/tiled_ensemble/components/smoothing.py

+        tiler (EnsembleTiler): Tiler object used to get tile dimension data.
+    """
+
+    name = "pipeline"


How about we rename this to SmoothSeams?

ashwinvaidya17 · 2024-06-26T08:33:04Z

src/anomalib/pipelines/tiled_ensemble/train_pipeline.py

+            )
+        runners.append(SerialRunner(MergeJobGenerator()))
+
+        if args["pipeline"]["ensemble"]["post_processing"]["seam_smoothing"]["apply"]:


Since each job is supposed to be independent, how about we move this to a separate section in the config.
Instead of defining the entire config under pipeline we can move each config under its separate key. So, if we set the SmoothingJob.name parameter to SmoothSeams or something, we can then move seam_smoothing section under post_processing to SmoothSeams.

TrainModel: ... Predict: ... SmoothSeams: apply: True sigma: 2 width: 0.1 ComputeStatistics: ...

Then we can just check if args['SmoothSeams']['apply']:

The solution above should work for this case and is much nicer than what I have now.

The problem here is that some pipeline stages require args from some other part of config. For example, model training job needs to know normalization stage to determine if normalization should be applied at tile level, but normalization stage is is part of post processing. Since the pipeline class does this _args = args.get(runner.generator.job_class.name, None) it would then mean that train job doesn't have access, or I'd need to duplicate this.

So I'm not sure how to best handle such cases with current design.

One possibility is to somehow accumulate the ones that are shared under one name, but then the config file looses the structure a bit.

ashwinvaidya17 · 2024-06-26T08:35:14Z

src/anomalib/pipelines/tiled_ensemble/components/smoothing.py

+        # tiler is used to determine where seams appear
+        tiler = get_ensemble_tiler(args)
+        yield SmoothingJob(
+            accelerator=args["accelerator"],


How about we pass accelerator in the init method? This way, the only arguments this method relies are under the seam_smoothing method.

That could work yeah. Seam smoothing actually relies only on the params under seam_smoothing.

ashwinvaidya17 · 2024-06-26T08:53:07Z

src/anomalib/pipelines/tiled_ensemble/components/smoothing.py

+            accelerator=args["accelerator"],
+            predictions=prev_stage_result,
+            width_factor=args["ensemble"]["post_processing"]["seam_smoothing"]["width"],
+            filter_sigma=args["ensemble"]["post_processing"]["seam_smoothing"]["width"],


Is this supposed to be sigma?

Yes, thanks.

ashwinvaidya17 · 2024-06-26T09:19:17Z

tests/pre_merge/tools/tiled_ensemble/conftest.py

@@ -0,0 +1,107 @@
+"""Fixtures that are used in tiled ensemble testing"""
+
+# Copyright (C) 2023 Intel Corporation


Suggested change

# Copyright (C) 2023 Intel Corporation

# Copyright (C) 2024 Intel Corporation

ashwinvaidya17 · 2024-06-26T09:21:13Z

tools/tiled_ensemble/ensemble_functions.py

+from pytorch_lightning.callbacks import ModelCheckpoint
+from tools.tiled_ensemble.ensemble_tiler import EnsembleTiler
+from tools.tiled_ensemble.post_processing.postprocess import NormalizationStage
+from tools.tiled_ensemble.predictions import (


Is this up-to-date?

These are old files, will be removed.

ashwinvaidya17 · 2024-06-26T09:22:02Z

tools/tiled_ensemble/ensemble_tiler.py

@@ -0,0 +1,112 @@
+"""Tiler used with ensemble of models."""


Do we need this here now that is moved to src/anomalib?

No, I'll delete this.

ashwinvaidya17 · 2024-06-26T09:23:12Z

tools/tiled_ensemble/test_ensemble.py

@@ -0,0 +1,124 @@
+"""Anomalib Testing Script for ensemble of models.


I guess this needs to be updated as well.

I'll go over the test and rewrite them after I refactor the other things that you pointed out.

ashwinvaidya17 · 2024-06-26T09:25:05Z

tools/tiled_ensemble/post_processing/__init__.py

@@ -0,0 +1,33 @@
+"""


I am confused about this. Is this still relevant?

No it's not, I'll remove this.

blaz-r · 2024-06-26T10:55:10Z

Okay. I'll check those out, thanks for going over the code. Indeed there are still some files left over as the refactor is still not 100% done therefore I left it as draft PR). I still need to cover the tests and all so I didn't delete everything.

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

blaz-r · 2024-08-02T16:44:42Z

I answered to some comments, and will refactor the config as you suggested in on of the comments. After the refactors I'll also update/rewrite all the tests. Thanks for all the feedback and patience, I'm quite busy right now and I'll really try to get this sorted when I get some time here and there.

blaz-r and others added 30 commits March 18, 2023 17:34

Fixed broken links in readme

73fa062

Fixed inference command in readme

eeb0b90

Merge branch 'openvinotoolkit:main' into main

4c60ab7

Merge branch 'openvinotoolkit:main' into main

b38ea6d

Merge branch 'openvinotoolkit:main' into main

7ea1047

Merge branch 'openvinotoolkit:main' into main

24d32f8

Add tiling for ensemble

971cd7f

Add tests for tiling for ensemble

53d110b

Moved ensemble tiler to separate file

3237379

Modify padim config for ensemble

621e1b4

Add tiling to dataset

2c7785c

Revert changes to train

b934db3

Add tiling to collate fn

71dabaf

Fix tiling in collate

ef69183

Merge branch 'openvinotoolkit:main' into ensemble

6c1357c

Change val. function to protected

0845114

Add tile number logic

7a0ecfa

Move collate fn to separate file

8437875

Update tests for tiler

c8ecb6e

Add training loop for ensemble

86c62c7

Add model input size setup

f9bb615

Move ens config to separate file

3e2dbda

Revert mvtec modifications

28ea8a2

Remove unused imports in mvtec

c321e60

Add batch adjustment to untiling

cd90ef3

Add predict step to ensemble

941439e

Add comment and docstring to tile joining function

42023c6

Move tile joining to separate function

69bd0e8

Add joining for all tiled data

06eb042

Add joining for all box data

67b9c3a

blaz-r added 2 commits June 12, 2024 22:02

Make collate function a datamodule attribute

bb04a3b

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor tiled ensemble train into pipeline step

658ee60

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

blaz-r marked this pull request as draft June 12, 2024 21:16

blaz-r added 18 commits June 13, 2024 14:11

Refactor tiled ensemble prediction into pipeline step

aecf837

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor tiled ensemble merging into pipeline step

15eae34

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor tiled ensemble seam smoothing into pipeline step

c00f6a1

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor tiled stats calculation into pipeline step

f7ee730

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Fix ckpt loading when predicting on test set.

9d1f141

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Add logging and add tqdm to pipeline steps.

6570156

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor normalization pipeline step

e49dbfe

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Refactor thresholding into new pipeline job

affc8ef

Fix transforms issue when predicting with dataloader

b934c68

Add visualization as new pipeline step

c0791be

Add metrics as new pipeline step

0dac5ed

Format the code and address some lint problems

fedaddb

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Add code to skip test if test split is none

3548a50

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Add accelerator to metrics and smoothing

551d38d

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Make threshold acq helper function and add to threshold to metrics

d6834d8

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Make a separate test pipeline

2b45cd2

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Restructure tiled ensemble files into directories

9713112

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

Pipeline code cleanup

dac985f

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

ashwinvaidya17 added this to the v1.2.0 milestone Jun 26, 2024

ashwinvaidya17 reviewed Jun 26, 2024

View reviewed changes

samet-akcay mentioned this pull request Jul 25, 2024

Tile with PatchCore OutOfMemoryError: CUDA out of memory. #2207

Closed

blaz-r and others added 4 commits August 2, 2024 17:44

Merge branch 'openvinotoolkit:main' into ensemble

b59fcbf

Remove old tiled ensemble files

aca61df

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Remove old post processing files

27824df

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Fix sigma value read in smoothing

0f935a7

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ensembling methods for tiling to Anomalib #1226

Add ensembling methods for tiling to Anomalib #1226

blaz-r commented Aug 1, 2023 •

edited

Loading

ashwinvaidya17 left a comment

ashwinvaidya17 Sep 11, 2023

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024 •

edited

Loading

ashwinvaidya17 Jun 26, 2024

blaz-r Aug 2, 2024

blaz-r commented Jun 26, 2024 •

edited

Loading

blaz-r commented Aug 2, 2024

		@@ -0,0 +1,107 @@
		"""Fixtures that are used in tiled ensemble testing"""

		# Copyright (C) 2023 Intel Corporation

	# Copyright (C) 2023 Intel Corporation
	# Copyright (C) 2024 Intel Corporation

		@@ -0,0 +1,124 @@
		"""Anomalib Testing Script for ensemble of models.

Add ensembling methods for tiling to Anomalib #1226

Are you sure you want to change the base?

Add ensembling methods for tiling to Anomalib #1226

Conversation

blaz-r commented Aug 1, 2023 • edited Loading

Description

Changes

Checklist

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaz-r Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaz-r commented Jun 26, 2024 • edited Loading

blaz-r commented Aug 2, 2024

blaz-r commented Aug 1, 2023 •

edited

Loading

blaz-r Aug 2, 2024 •

edited

Loading

blaz-r commented Jun 26, 2024 •

edited

Loading