OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.

[cdeepali]

Executing test_jit tests with OpenCE is showing numerous warnings:

OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.

This same warning is not observed with WMLCE for same version of OpenBLAS.

[cdeepali]

As a workaround user can set OMP_NUM_THREADS=1 to avoid using multiple threads.

[cdeepali]

pytorch/pytorch#52047

[lgg]

@cdeepali i had the same issue. Building OpenBLAS from source with USE_OPENMP=1 didn't help for me.

I have few Jetson Nano and I guess i found a few ways to fix it:

1. Downgrading Jetpack version
   On Jetson Nano with JetPack 4.4.1 - everything works fine, on new Jeston Nano with JetPack 4.5.1 - i have the same issue.
   Check the difference from jtop output:

Jetpack 4.4.1

Jetpack 4.5.1

Nvidia update VPI (https://docs.nvidia.com/vpi/index.html) and i guess it cause the problem.

2. Downgrading torch
   On another Jetson Nano with Jetpack 4.5.1 i tried this:

• sudo apt-get install libopenblas-base libopenmpi-dev
• wget https://nvidia.box.com/shared/static/wa34qwrwtk9njtyarwt5nvo6imenfy26.whl -O torch-1.7.0-cp36-cp36m-linux_aarch64.whl
• ./venv/bin/pip3 install torch-1.7.0-cp36-cp36m-linux_aarch64.whl
• and it helps

[bnemanich]

This was resolved by switching PyTorch to use TBB instead of OpenMP.

[lgg]

@bnemanich how did you do that?

[cdeepali]

We switched to using TBB for intra-op parallelism. - https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html#:~:text=PyTorch%20uses%20a%20single%20thread,(%20intra%2Dop%20parallelism%20).

[jayfurmanek]

yeah, the change was in:
2ab8570

[jayfurmanek]

And just a note for why this was needed.

Anaconda's toolchain uses gomp for openMP. And gomp is known to not be fork-safe (it can hang when the program using it forks). To avoid any hangs, Anaconda has disabled openMP in their OpenBLAS package. (conda-forge has since included llvm-openmp as a openmp variant, but defaults hasn't moved there yet)

This warning message comes when Pytorch is built with OpenMP support, but then uses an OpenBLAS that was not.

Switching Pytorch to use TBB instead of OMP avoids the problem.

The downside is still relying on the OpenBLAS package from defaults with no OMP support, which is slower. In a future release we will likely include our own OpenBLAS package built with llvm-openmp unless defaults provides.