Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PaDiM: model outputs NaN when inferencing images sequencially #385

Closed
tsmiyamoto opened this issue Jun 23, 2022 · 12 comments · Fixed by #392
Closed

PaDiM: model outputs NaN when inferencing images sequencially #385

tsmiyamoto opened this issue Jun 23, 2022 · 12 comments · Fixed by #392
Labels
Inference Model

Comments

@tsmiyamoto
Copy link

Describe the bug

  • I'm planning to do inference on webcam input. The inference works well on the first frame, but on the second frame, it doesn't.
  • Here're the image outputs and console outputs (print(predictions))

Code (webcam input)

from anomalib.data.utils import read_image
from anomalib.post_processing import compute_mask, superimpose_anomaly_map
from anomalib.post_processing.normalization.cdf import normalize as normalize_cdf
from anomalib.post_processing.normalization.cdf import standardize
from anomalib.post_processing.normalization.min_max import (
    normalize as normalize_min_max,
)
from anomalib.deploy.inferencers.base import Inferencer
import cv2
from importlib import import_module
from argparse import ArgumentParser, Namespace
from anomalib.config import get_configurable_parameters
import warnings
from pathlib import Path


def get_args() -> Namespace:
    parser = ArgumentParser()
    parser.add_argument("--config", type=Path, required=True, help="Path to a model config file")
    parser.add_argument("--weight_path", required=True)
    # parser.add_argument("--meta_data", default="")
    args = parser.parse_args()
    return args    


def main():
    args = get_args()
    config = get_configurable_parameters(config_path=args.config)

    cap = cv2.VideoCapture(0)

    inferencer: Inferencer
    module = import_module("anomalib.deploy.inferencers.torch")
    TorchInferencer = getattr(module, "TorchInferencer")
    inferencer = TorchInferencer(config=config, model_source=args.weight_path, meta_data_path=None)
    if hasattr(inferencer, "meta_data"):
        meta_data = getattr(inferencer, "meta_data")
    else:
        meta_data = {}

    while(True):
        ret, frame = cap.read()
        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        meta_data["image_shape"] = image.shape[:2]

        processed_image = inferencer.pre_process(image)
        predictions = inferencer.forward(processed_image)
        print(predictions)
        anomaly_map, pred_scores = inferencer.post_process(predictions, meta_data=meta_data)
        print(f"Anomaly_score: {pred_scores}")

        # Overlay segmentation mask using raw predictions
        if meta_data is not None:
            image = inferencer._superimpose_segmentation_mask(meta_data, anomaly_map, image)
        anomaly_map = superimpose_anomaly_map(anomaly_map, image)

        cv2.imshow("Anomaly map", anomaly_map)
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

    cap.release()
    cv2.destroyAllWindows()

if __name__ == "__main__":
    main()

Code output

tensor([[[[41.7540, 41.4636, 40.6263,  ..., 34.3366, 34.7471, 34.8892],
          [41.5304, 41.2426, 40.4131,  ..., 34.1754, 34.5826, 34.7236],
          [40.8823, 40.6022, 39.7947,  ..., 33.7111, 34.1088, 34.2464],
          ...,
          [19.1933, 19.1503, 19.0268,  ..., 18.9357, 19.0051, 19.0278],
          [19.4248, 19.3804, 19.2529,  ..., 19.1242, 19.1945, 19.2177],
          [19.5046, 19.4598, 19.3309,  ..., 19.1878, 19.2586, 19.2818]]]])
Anomaly_score: 1.0
tensor([[[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          ...,
          [18.6857, 18.6415, 18.5146,  ..., 18.1626, 18.2075, 18.2221],
          [18.9330, 18.8873, 18.7560,  ..., 18.3440, 18.3900, 18.4050],
          [19.0184, 18.9721, 18.8394,  ..., 18.4052, 18.4517, 18.4668]]]])
Anomaly_score: nan
tensor([[[[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]]]])
Anomaly_score: nan

Expected behavior

  • A clear and concise description of what you expected to happen.

Screenshots
1st input
000

2nd input
001

Hardware and Software Configuration

  • Hardware: NVIDIA Jetson Nano
  • OS: Ubuntu18.04 (JetPack 4.6.1)
  • CUDA Version: 11.4.166
  • Docker: nvcr.io/nvidia/l4t-pytorch:r34.1.1-pth1.11-py3

Additional context

  • This happens when running on WSL2.
@samet-akcay
Copy link
Contributor

wow, that's a pretty nasty bug :D Thanks for reporting it @tsmiyamoto, we'll check it out

@tsmiyamoto
Copy link
Author

@samet-akcay
Thank you for your reply!
We'll work on this issue, too.

@tsmiyamoto
Copy link
Author

tsmiyamoto commented Jun 24, 2022 •
edited
Loading

Sorry, this is my misunderstanding.
It's working on WSL2.
Might be the issue regarding Jetson?

This also occurs when executing inference on images in directory.

stream() in inference.py
https://github.com/openvinotoolkit/anomalib/blob/development/tools/inference.py

@VdLMV
Copy link
Contributor

VdLMV commented Jun 25, 2022

I had similar problems. Maybe this is of help to you.
Source of problem was 'torch.sqrt(distances)' in AnomalyMapGenerator.compute_distance. With custom dataset and backbone distances sometimes contains small negative numbers, possible due to imperfect pseudo inverse calculation. Solution was to clip distances to zero before sqrt.

@samet-akcay
Copy link
Contributor

@VdLMV, great find! Thanks a lot! Would you like to make changes to become a contributor, or would you prefer us to fix it instead?

@VdLMV
Copy link
Contributor

VdLMV commented Jun 28, 2022

I will issue a PR later this week.

@samet-akcay samet-akcay linked a pull request Jul 11, 2022 that will close this issue
PaDim occasionally NaNs in anomaly map #392
Merged
11 tasks
@samet-akcay
Copy link
Contributor

#392 aims to address this issue. Since it was merged, I'm closing this issue. Feel free to re-open if you still experience the same behaviour.

@tsmiyamoto
Copy link
Author

@VdLMV Thank you for fixing it!!!

@samet-akcay Thank you for your reply. This PaDiM repository is really great 👍

@beniben0
Copy link

Describe the bug

  • I'm planning to do inference on webcam input. The inference works well on the first frame, but on the second frame, it doesn't.
  • Here're the image outputs and console outputs (print(predictions))

Code (webcam input)

from anomalib.data.utils import read_image
from anomalib.post_processing import compute_mask, superimpose_anomaly_map
from anomalib.post_processing.normalization.cdf import normalize as normalize_cdf
from anomalib.post_processing.normalization.cdf import standardize
from anomalib.post_processing.normalization.min_max import (
    normalize as normalize_min_max,
)
from anomalib.deploy.inferencers.base import Inferencer
import cv2
from importlib import import_module
from argparse import ArgumentParser, Namespace
from anomalib.config import get_configurable_parameters
import warnings
from pathlib import Path


def get_args() -> Namespace:
    parser = ArgumentParser()
    parser.add_argument("--config", type=Path, required=True, help="Path to a model config file")
    parser.add_argument("--weight_path", required=True)
    # parser.add_argument("--meta_data", default="")
    args = parser.parse_args()
    return args    


def main():
    args = get_args()
    config = get_configurable_parameters(config_path=args.config)

    cap = cv2.VideoCapture(0)

    inferencer: Inferencer
    module = import_module("anomalib.deploy.inferencers.torch")
    TorchInferencer = getattr(module, "TorchInferencer")
    inferencer = TorchInferencer(config=config, model_source=args.weight_path, meta_data_path=None)
    if hasattr(inferencer, "meta_data"):
        meta_data = getattr(inferencer, "meta_data")
    else:
        meta_data = {}

    while(True):
        ret, frame = cap.read()
        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        meta_data["image_shape"] = image.shape[:2]

        processed_image = inferencer.pre_process(image)
        predictions = inferencer.forward(processed_image)
        print(predictions)
        anomaly_map, pred_scores = inferencer.post_process(predictions, meta_data=meta_data)
        print(f"Anomaly_score: {pred_scores}")

        # Overlay segmentation mask using raw predictions
        if meta_data is not None:
            image = inferencer._superimpose_segmentation_mask(meta_data, anomaly_map, image)
        anomaly_map = superimpose_anomaly_map(anomaly_map, image)

        cv2.imshow("Anomaly map", anomaly_map)
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

    cap.release()
    cv2.destroyAllWindows()

if __name__ == "__main__":
    main()

Code output

tensor([[[[41.7540, 41.4636, 40.6263,  ..., 34.3366, 34.7471, 34.8892],
          [41.5304, 41.2426, 40.4131,  ..., 34.1754, 34.5826, 34.7236],
          [40.8823, 40.6022, 39.7947,  ..., 33.7111, 34.1088, 34.2464],
          ...,
          [19.1933, 19.1503, 19.0268,  ..., 18.9357, 19.0051, 19.0278],
          [19.4248, 19.3804, 19.2529,  ..., 19.1242, 19.1945, 19.2177],
          [19.5046, 19.4598, 19.3309,  ..., 19.1878, 19.2586, 19.2818]]]])
Anomaly_score: 1.0
tensor([[[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
          ...,
          [18.6857, 18.6415, 18.5146,  ..., 18.1626, 18.2075, 18.2221],
          [18.9330, 18.8873, 18.7560,  ..., 18.3440, 18.3900, 18.4050],
          [19.0184, 18.9721, 18.8394,  ..., 18.4052, 18.4517, 18.4668]]]])
Anomaly_score: nan
tensor([[[[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]]]])
Anomaly_score: nan

Expected behavior

  • A clear and concise description of what you expected to happen.

Screenshots 1st input 000

2nd input 001

Hardware and Software Configuration

  • Hardware: NVIDIA Jetson Nano
  • OS: Ubuntu18.04 (JetPack 4.6.1)
  • CUDA Version: 11.4.166
  • Docker: nvcr.io/nvidia/l4t-pytorch:r34.1.1-pth1.11-py3

Additional context

  • This happens when running on WSL2.

@tsmiyamoto which version of Python did you use?
Thank you for your reply

@tsmiyamoto
Copy link
Author

@beniben0
Thank you for your question.
I was using Python 3.6.9, which is pre installed on Jetson Nano (JetPack 4.6.1).

@beniben0
Copy link

beniben0 commented Aug 12, 2022 •
edited
Loading

@beniben0 Thank you for your question. I was using Python 3.6.9, which is pre installed on Jetson Nano (JetPack 4.6.1).

Good to know thanks. I thought anomalib required version 3.8.

Another question, which pytorch version do you use?
I'm struggling with installing anomalib on my Jetson Xavier NX.

@jasonrichdarmawan
Copy link

jasonrichdarmawan commented Feb 5, 2023 •
edited
Loading

@beniben0 Thank you for your question. I was using Python 3.6.9, which is pre installed on Jetson Nano (JetPack 4.6.1).

May I know how you do that? I can't install it with Python 3.6.9 nor Python3.8.0 (psutil is not available on aarch64 architecture which is used by NVIDIA Jetson Nano B01)

Installing build dependencies ...   Getting requirements to build wheel ...   ERROR: Command errored out with exit status 1:
   command: [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python) [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py) get_requires_for_build_wheel [/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7](https://file+.vscode-resource.vscode-cdn.net/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7)
       cwd: [/Volumes/T7Touch/Work/GoDataID/anomalib](https://file+.vscode-resource.vscode-cdn.net/Volumes/T7Touch/Work/GoDataID/anomalib)
  Complete output (17 lines):
  Traceback (most recent call last):
    File "[/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py)", line 349, in <module>
      main()
    File "[/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py)", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "[/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py)", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "[/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py](https://file+.vscode-resource.vscode-cdn.net/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py)", line 163, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "[/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py](https://file+.vscode-resource.vscode-cdn.net/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py)", line 143, in _get_build_requires
      self.run_setup()
    File "[/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py](https://file+.vscode-resource.vscode-cdn.net/private/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/pip-build-env-nidn3j5c/overlay/lib/python3.6/site-packages/setuptools/build_meta.py)", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 6
      from __future__ import annotations
      ^
  SyntaxError: future feature annotations is not defined
  ----------------------------------------
WARNING: Discarding file:///Volumes/T7Touch/Work/GoDataID/anomalib. Command errored out with exit status 1: [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python) [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py) get_requires_for_build_wheel [/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7](https://file+.vscode-resource.vscode-cdn.net/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7) Check the logs for full command output.
ERROR: Command errored out with exit status 1: [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/bin/python) [/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py](https://file+.vscode-resource.vscode-cdn.net/Users/jason/opt/anaconda3/envs/GoDataID_anomalib037_python369/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py) get_requires_for_build_wheel [/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7](https://file+.vscode-resource.vscode-cdn.net/var/folders/w9/5lrd8m1x4f586v9ynn8xrryw0000gn/T/tmp02e4tpp7) Check the logs for full command output.
Note: you may need to restart the kernel to use updated packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Inference Model
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

PaDim occasionally NaNs in anomaly map
5 participants
@samet-akcay @beniben0 @jasonrichdarmawan @VdLMV @tsmiyamoto