Assertion at src/lib/jit/detail/executable_buffer.cpp:44

[ErfolgreichCharismatisch]

Describe the bug
Assertion at src/lib/jit/detail/executable_buffer.cpp:44

Environment
Include all relevant environment information:

1. OS [e.g. Ubuntu 18.04]: Ubuntu 18.04.6 LTS
2. Python version [e.g. 3.7]: Python 3.8.0
3. DeepSparse version or commit hash [e.g. 0.1.0, f7245c8]: 7d31c4bf
4. ML framework version(s) [e.g. torch 1.7.1]: torch 1.10.2+cu102
5. Other Python package versions [e.g. SparseML, Sparsify, numpy, ONNX]: numpy 1.21.6, onnx 1.12.0
6. CPU info - output of deepsparse/src/deepsparse/arch.bin or output of cpu_architecture() as follows:

>>> import deepsparse.cpu
>>> print(deepsparse.cpu.cpu_architecture())

/home/umgebung/lib/python3.8/site-packages/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "

{'L1_data_cache_size': 32768, 'L1_instruction_cache_size': 32768, 'L2_cache_size': 262144, 'L3_cache_size': 31457280, 'architecture': 'x86_64', 'available_cores_per_socket': 6, 'available_num_cores': 6, 'available_num_hw_threads': 6, 'available_num_numa': 1, 'available_num_sockets': 1, 'available_sockets': 1, 'available_threads_per_core': 1, 'bf16': False, 'cores_per_socket': 6, 'dotprod': False, 'i8mm': False, 'isa': 'avx2', 'num_cores': 6, 'num_hw_threads': 6, 'num_numa': 1, 'num_sockets': 1, 'threads_per_core': 1, 'vendor': 'GenuineIntel', 'vendor_id': 'Intel', 'vendor_model': 'Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz', 'vnni': False}

To Reproduce
In a virtual env umgebung

(umgebung) root@server:/home/umgebung# deepsparse.benchmark zoo:nlp/question_answering/bert-base/pytorch/huggingface/squad/pruned95_obs_quant-none

Errors
Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
1# wand::detail::assert_fail(char const*, char const*, int) in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
2# 0x00007F8E50FD997C in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
3# 0x00007F8E50FB0CC0 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
4# 0x00007F8E50FB1368 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
5# 0x00007F8E50FB43B5 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
6# 0x00007F8E50DF0E85 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
7# 0x00007F8E51210FED in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
8# 0x00007F8E5125B0A3 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
9# 0x00007F8E51248111 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
10# 0x00007F8E5112C417 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
11# 0x00007F8E5112F7FB in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
12# 0x00007F8E511585E3 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
13# 0x00007F8E51158986 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
14# 0x00007F8E5106D2CB in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
15# 0x00007F8E50EEEEDD in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
16# 0x00007F8E50EEF172 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
17# 0x00007F8E50EEF323 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
18# 0x00007F8E50EEF7A8 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
19# 0x00007F8E50EEF810 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
20# 0x00007F8E5034DD27 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
21# 0x00007F8E5034E22F in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
22# 0x00007F8E502C1AD5 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
23# 0x00007F8E502B1049 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0

Additional context
2023-01-27 12:18:28 deepsparse.benchmark.benchmark_model INFO num_streams default value chosen of 3. This requires tuning and may be sub-optimal
INFO:deepsparse.benchmark.benchmark_model:num_streams default value chosen of 3. This requires tuning and may be sub-optimal
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.3.2 COMMUNITY | (7d31c4bf) (release) (optimized) (system=avx2, binary=avx2)
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.3.2 (7d31c4bf) (release) (optimized) (system=avx2, binary=avx2)
Date: 01-27-2023 @ 12:18:33 CET
OS: Linux server 4.15.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021
Arch: x86_64
ISA: x86_64-avx2-32-64-16-64-xbyak
CPU: GenuineIntel
Vendor: Intel
Cores/sockets/threads: [6, 1, 6]
Available cores/sockets/threads: [6, 1, 6]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 30Mb
Total memory: 16G
Free memory: 8.03798G

[tlrmchlsmth]

@ErfolgreichCharismatisch thank you for the bug report. Our team is looking into it now.

To make it easier for us to debug, could you share the output of lscpu and numactl -H?

[ErfolgreichCharismatisch]

Architektur: x86_64
CPU Operationsmodus: 32-bit, 64-bit
Byte-Reihenfolge: Little Endian
CPU(s): 6
Liste der Online-CPU(s): 0-5
Thread(s) pro Kern: 1
Kern(e) pro Socket: 6
Sockel: 1
Anbieterkennung: GenuineIntel
Prozessorfamilie: 6
Modell: 63
Modellname: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping: 2
CPU MHz: 1999.918
BogoMIPS: 4994.22
Virtualisierung: VT-x
Hypervisor-Anbieter: Parallels
Virtualisierungstyp: Container
L1d Cache: 32K
L1i Cache: 32K
L2 Cache: 256K
L3 Cache: 30720K
Markierungen: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu cpuid_faulting pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d

numactl -H wasn't found.

[ErfolgreichCharismatisch]

Did not change.

(umgebung) root@server:/home/umgebung# export NM_JIT_CACHE_STRIPING=1
(umgebung) root@server:/home/umgebung# deepsparse.benchmark zoo:nlp/question_answering/bert-base/pytorch/huggingface/squad/pruned95_obs_quant-none
/home/umgebung/lib/python3.8/site-packages/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
2023-02-01 16:19:12 deepsparse.benchmark.benchmark_model INFO Thread pinning to cores enabled
2023-02-01 16:19:13 deepsparse.benchmark.benchmark_model INFO num_streams default value chosen of 3. This requires tuning and may be sub-optimal
INFO:deepsparse.benchmark.benchmark_model:num_streams default value chosen of 3. This requires tuning and may be sub-optimal
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.3.2 COMMUNITY | (7d31c4bf) (release) (optimized) (system=avx2, binary=avx2)
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.3.2 (7d31c4bf) (release) (optimized) (system=avx2, binary=avx2)
Date: 02-01-2023 @ 16:19:16 CET
OS: Linux server 4.15.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021
Arch: x86_64
ISA: x86_64-avx2-32-64-16-64-xbyak
CPU: GenuineIntel
Vendor: Intel
Cores/sockets/threads: [6, 1, 6]
Available cores/sockets/threads: [6, 1, 6]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 30Mb
Total memory: 16G
Free memory: 10.101G

Assertion at src/lib/jit/detail/executable_buffer.cpp:44

Backtrace:
0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
1# wand::detail::assert_fail(char const*, char const*, int) in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
2# 0x00007F85FC2ED97C in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
3# 0x00007F85FC93D606 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
4# 0x00007F85FC93DE7A in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
5# 0x00007F85FC93E49A in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
6# 0x00007F85FC743B34 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
7# 0x00007F85FC748BF7 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
8# 0x00007F85FC3E7303 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
9# 0x00007F85FC3E788B in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
10# 0x00007F85FC3E7F23 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
11# 0x00007F85FC3E817C in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
12# 0x00007F85FC74478B in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
13# 0x00007F85FC3C910A in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
14# 0x00007F85FC3C9622 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
15# 0x00007F85FC3CD3E9 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
16# 0x00007F85FC3C7B2F in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
17# 0x00007F85FC38058E in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
18# 0x00007F85FC202EDD in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
19# 0x00007F85FC203172 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
20# 0x00007F85FC203323 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
21# 0x00007F85FC2037A8 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
22# 0x00007F85FC203810 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
23# 0x00007F85FB661D27 in /home/umgebung/lib/python3.8/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0

[ErfolgreichCharismatisch]

Push.

[bnellnm]

@ErfolgreichCharismatisch , this should be fixed in the latest nightly release.

[ErfolgreichCharismatisch]

Yes, it was, well done.

[bnellnm]

Great, thanks for checking the nightly!