(WIP) Visualizer improvements pt2 #298

djdameln · 2022-05-06T05:55:37Z

Description

(WIP) Second part of the visualizer refactor. This PR mainly focuses on the inferencer. The torch inferencer was simplified to use PyTorch Lightning. The visualizer callback was refactored to also generate and show the visualization results after each predict batch. OpenVINO inference was moved to a separate file in tools/openvino.py

Addresses #317 #294 #326

Known issues:
Currently the visualization results cannot be customized (which types of images should be shown)
Visualization results are only shown, not saved to the file system.

Changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

My code follows the pre-commit style and check guidelines of this project.
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing tests pass locally with my changes

ashwinvaidya17

This is a nice modification. I am looking forward for the final inferencer design

ashwinvaidya17 · 2022-05-10T08:04:26Z

tools/openvino.py

+    """
+    parser = ArgumentParser()
+    # --model_config_path will be deprecated in 0.2.8 and removed in 0.2.9
+    parser.add_argument("--model_config_path", type=str, required=False, help="Path to a model config file")


Since this is a new file, I feel we should drop deprecated flags.

ashwinvaidya17 · 2022-05-10T08:05:41Z

tools/openvino.py

+    parser.add_argument("--weight_path", type=Path, required=True, help="Path to a model weights")
+    parser.add_argument("--image_path", type=Path, required=True, help="Path to an image to infer.")
+    parser.add_argument("--save_path", type=Path, required=False, help="Path to save the output image.")
+    parser.add_argument("--meta_data", type=Path, required=False, help="Path to JSON file containing the metadata.")


We need metadata to load threshold for OpenVINO models. required should be set to True.

ashwinvaidya17 · 2022-05-10T08:06:09Z

tools/openvino.py

+    )
+
+    args = parser.parse_args()
+    if args.model_config_path is not None:


Same here. We don't need deprecation warnings in a new file.

ashwinvaidya17 · 2022-05-10T08:11:38Z

anomalib/utils/callbacks/visualizer_callback.py

+            _dataloader_idx (int): Index of the dataloader that yielded the current batch (unused).
+        """
+        for visualizer in self.generate_visualizer(outputs):
+            visualizer.show()


Since this is work in progress I am assuming the final design will not always show the results by default.

This requires substantial changes to the visualizer callback class, so I'm planning to address that in part 3 of the refactor. That one will focus on customizability of the visualizations. This means that if we merge this PR, we will temporarily always show the visualizations during inference. If we don't want this, I can re-target this PR to a feature branch to which I'll then also target pt3.

ashwinvaidya17 · 2022-05-10T08:14:37Z

anomalib/utils/callbacks/visualizer_callback.py

@@ -96,6 +95,60 @@ def _add_images(
        if "local" in module.hparams.project.log_images_to:
            visualizer.save(Path(module.hparams.project.path) / "images" / filename.parent.name / filename.name)

+    def generate_visualizer(self, outputs):


This is a nice use of yield. I think we should also add a return type to the method. Something like -> Iterator[Visualizer]

samet-akcay · 2022-05-12T15:45:36Z

anomalib/deploy/inferencers/torch.py

+        warnings.warn(
+            "Torch Inferencer is deprecated and will be removed in a future version. To run Inference with a PyTorch "
+            "model, please follow the example in `tools/inference.py`"
+        )


maybe we could decide and specify when we deprecate it. Note that, in every sprint, we aim to release a new version.

Actually, I'm not entirely sure if we should deprecate this file at all. There might be some use cases where users want to perform pure pytorch inference. What do you think @samet-akcay @ashwinvaidya17?

Do we support pure pytorch inference? If I remember correctly we still load the model config file for pytorch models and load the lightning module.

Hmm yeah, we still use the PL module in this (the old) torch inferencer. In that case, maybe we could refactor the (pure) torch inferencer to load the model from a .pth file and only use the classes in torch_model.py.

I feel we should support the possibility of pure pytorch inference, in case some users might need it. Default could still be the new PL inferencer.

If we decide to keep the torch implementation, this would then need to be removed

samet-akcay · 2022-05-12T15:59:51Z

tools/openvino.py

+def add_label(prediction: np.ndarray, scores: float, font: int = cv2.FONT_HERSHEY_PLAIN) -> np.ndarray:
+    """If the model outputs score, it adds the score to the output image.
+
+    Args:
+        prediction (np.ndarray): Resized anomaly map.
+        scores (float): Confidence score.
+
+    Returns:
+        np.ndarray: Image with score text.
+    """
+    text = f"Confidence Score {scores:.0%}"
+    font_size = prediction.shape[1] // 1024 + 1  # Text scale is calculated based on the reference size of 1024
+    (width, height), baseline = cv2.getTextSize(text, font, font_size, thickness=font_size // 2)
+    label_patch = np.zeros((height + baseline, width + baseline, 3), dtype=np.uint8)
+    label_patch[:, :] = (225, 252, 134)
+    cv2.putText(label_patch, text, (0, baseline // 2 + height), font, font_size, 0, lineType=cv2.LINE_AA)
+    prediction[: baseline + height, : baseline + width] = label_patch
+    return prediction


Does this address PR #291?

samet-akcay · 2022-05-12T16:01:10Z

tools/openvino.py

+    module = import_module("anomalib.deploy.inferencers.openvino")
+    OpenVINOInferencer = getattr(module, "OpenVINOInferencer")  # pylint: disable=invalid-name


Is this still needed? Wouldn't it be possible to just import OpenVINOInferencer at the top of the module?

samet-akcay · 2022-05-12T16:03:12Z

tools/openvino.py

+    if args.image_path.is_dir():
+        # Write the output to save_path in the same structure as the input directory.
+        for image in args.image_path.glob("**/*"):
+            if image.is_file() and image.suffix in (".jpg", ".png", ".jpeg"):


Can we use IMG_ExTENSIONS from torchvision?

from torchvision.datasets.folder import IMG_EXTENSIONS

The reason being that there could be other image extensions we could actually support, but exclude with (".jpg", ".png", ".jpeg")

* add methods for adding labels to image * refactor visualizer * move add_label to post processing helpers * fix tests * support single image in visualizer * call generate * revert default task type * typing * refactor inferencer and visualizer * add torch inference entrypoint * revert torch inferencer for backward compatibility * add torch inferencer to init files * pause visualization * openvino.py -> openvino_inference.py * address comments * remove unused argument * remove depracation warning

* Visualizer improvements Part II (#298) * add methods for adding labels to image * refactor visualizer * move add_label to post processing helpers * fix tests * support single image in visualizer * call generate * revert default task type * typing * refactor inferencer and visualizer * add torch inference entrypoint * revert torch inferencer for backward compatibility * add torch inferencer to init files * pause visualization * openvino.py -> openvino_inference.py * address comments * remove unused argument * remove depracation warning * 📜 Add Inference Documentation (#308) * add inference and export guides * finish sentence * fix typo * add to index, change required field * 🚜 Visualizer refactor pt3 (#374) * make saving and showing of visualizations configurable * make saving and showing of visualizations configurable * fix visualizer callback tests * remove unused imports * fix openvino test * add comments * Visualizer refactor pt4 (#379) * make visualization mode configurable * change default visualization mode * value checks * check mask range, move save and show to visualizer * fix visualizer tests * change pred_mask value check * address circular import * address mypy issue * fix visualizer test and mypy * fix openvino tests * fix typo * add license header * add new inference parameters to guide * refactor inference entrypoint structure * fix visualizer in cli * remove param from kwargs * fix visualizer args in notebook * Fix torchvision version (#397) * 🛠 Fix torchvision version (#397) Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>

djdameln added 10 commits April 29, 2022 11:56

add methods for adding labels to image

bcabc3c

refactor visualizer

97fad83

move add_label to post processing helpers

bfbf6d1

fix tests

1981238

support single image in visualizer

2b654a5

call generate

f51be3b

revert default task type

6e19e8e

typing

6d6054d

refactor inferencer and visualizer

c677c07

add torch inference entrypoint

39fb30f

djdameln requested review from ashwinvaidya17 and samet-akcay May 6, 2022 05:55

merge development

1cc8ecd

ashwinvaidya17 reviewed May 10, 2022

View reviewed changes

djdameln mentioned this pull request May 11, 2022

📜 Add Inference Documentation #308

Merged

11 tasks

djdameln added 3 commits May 12, 2022 14:44

revert torch inferencer for backward compatibility

ed6cf0a

add torch inferencer to init files

1bf45ef

pause visualization

a6f86f4

samet-akcay reviewed May 12, 2022

View reviewed changes

openvino.py -> openvino_inference.py

0ef8832

This was referenced May 16, 2022

Add support for Inference with Torch Models on CPU-only Machines #317

Closed

Where can I save the segmentation results separately #294

Closed

Label Prediction with Confidence score #326

Closed

djdameln added 2 commits May 23, 2022 16:05

address comments

31c5724

remove unused argument

54ec278

samet-akcay added Visualization Inference Enhancement New feature or request labels May 24, 2022

djdameln added 2 commits May 24, 2022 17:32

merge development

5065e62

remove depracation warning

1d929b0

djdameln changed the base branch from development to feature/visualization-inference-refactor May 25, 2022 14:52

ashwinvaidya17 approved these changes May 25, 2022

View reviewed changes

samet-akcay approved these changes May 25, 2022

View reviewed changes

samet-akcay merged commit ef4190d into feature/visualization-inference-refactor May 25, 2022

samet-akcay deleted the da/inference-visualization branch May 25, 2022 14:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(WIP) Visualizer improvements pt2 #298

(WIP) Visualizer improvements pt2 #298

djdameln commented May 6, 2022 •

edited by samet-akcay

Loading

ashwinvaidya17 left a comment

ashwinvaidya17 May 10, 2022

ashwinvaidya17 May 10, 2022

ashwinvaidya17 May 10, 2022

ashwinvaidya17 May 10, 2022

djdameln May 23, 2022

ashwinvaidya17 May 10, 2022

samet-akcay May 12, 2022

djdameln May 23, 2022

ashwinvaidya17 May 23, 2022

djdameln May 23, 2022 •

edited

Loading

djdameln May 23, 2022 •

edited

Loading

samet-akcay May 24, 2022

samet-akcay May 12, 2022

samet-akcay May 12, 2022

samet-akcay May 12, 2022

samet-akcay May 12, 2022

		module = import_module("anomalib.deploy.inferencers.openvino")
		OpenVINOInferencer = getattr(module, "OpenVINOInferencer") # pylint: disable=invalid-name

(WIP) Visualizer improvements pt2 #298

(WIP) Visualizer improvements pt2 #298

Conversation

djdameln commented May 6, 2022 • edited by samet-akcay Loading

Description

Changes

Checklist

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djdameln May 23, 2022 • edited Loading

Choose a reason for hiding this comment

djdameln May 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djdameln commented May 6, 2022 •

edited by samet-akcay

Loading

djdameln May 23, 2022 •

edited

Loading

djdameln May 23, 2022 •

edited

Loading