Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Flaky test MKLDNN_BASE.MKLDNNSum #11998

Closed
marcoabreu opened this issue Aug 2, 2018 · 8 comments
Closed

Flaky test MKLDNN_BASE.MKLDNNSum #11998

marcoabreu opened this issue Aug 2, 2018 · 8 comments
Labels
Disabled test Flaky Test

Comments

@marcoabreu
Copy link
Contributor

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1319/pipeline
Cpp: MKLDNN+GPU

[0;32m[----------] �[m2 tests from MKLDNN_BASE
�[0;32m[ RUN      ] �[mMKLDNN_BASE.MKLDNNSum
format: 8, 33
format: 50
format: 8, 33
format: 50
format: 8, 33
format: 50
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: In Place Copy [279936] with Normal NDArray [279936]
Verifying: In Place Copy [279936] with MKLDNN NDArray [279936]
Verifying: In Place Copy [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: In Place Copy [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: In Place Copy [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: In Place Copy [34848] with Normal NDArray [34848]
Verifying: In Place Copy [34848] with MKLDNN NDArray [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: In Place Copy [96,2916] with Normal NDArray [96,2916]
Verifying: In Place Copy [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: In Place Copy [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: In Place Copy [96,2916] with MKLDNN NDArray [96,2916]
Verifying: In Place Copy [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: In Place Copy [96,363] with Normal NDArray [96,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: In Place Copy [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: In Place Copy [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: In Place Copy [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: In Place Copy [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: In Place Copy [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
unknown file: Failure
C++ exception with description "[00:17:20] /work/mxnet/src/ndarray/ndarray.cc:444: Check failed: delay_alloc 

Stack trace returned 10 entries:
[bt] (0) build/tests/mxnet_unit_tests(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0xf9c75c]
[bt] (1) build/tests/mxnet_unit_tests(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0xf9db08]
[bt] (2) build/tests/mxnet_unit_tests(mxnet::NDArray::Chunk::SetMKLMem(nnvm::TShape const&, int)+0x4ef) [0x455370f]
[bt] (3) build/tests/mxnet_unit_tests(mxnet::NDArray::GetMKLDNNData() const+0xa8) [0x45539c8]
[bt] (4) build/tests/mxnet_unit_tests(void mxnet::CopyFromToDnsImpl<mshadow::cpu, mshadow::cpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext)+0x2e8) [0x456e7e8]
[bt] (5) build/tests/mxnet_unit_tests(void mxnet::CopyFromToImpl<mshadow::cpu, mshadow::cpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&)+0x4d0) [0x45764a0]
[bt] (6) build/tests/mxnet_unit_tests() [0x455fdff]
[bt] (7) build/tests/mxnet_unit_tests(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x48d) [0x29c03bd]
[bt] (8) build/tests/mxnet_unit_tests(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0xe2) [0x29d9412]
[bt] (9) build/tests/mxnet_unit_tests(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run()+0x4a) [0x29bfe1a]

" thrown in the test body.
�[0;31m[  FAILED  ] �[mMKLDNN_BASE.MKLDNNSum (4679 ms)
@marcoabreu
Copy link
Contributor Author

@pengzhao-intel

@pengzhao-intel
Copy link
Contributor

@jinhuang415 please help take a look for this issue.

@anirudh2290 anirudh2290 mentioned this issue Aug 6, 2018
Merged
4 tasks
@lupesko
Copy link
Contributor

lupesko commented Aug 7, 2018

@jinhuang415 - bounce
@azai91 FYI

@jinhuang415
Copy link
Contributor

We have reproduced this issue in local luster, looks only happens very occasionally, will update once there is further findings.

@jinhuang415
Copy link
Contributor

jinhuang415 commented Aug 8, 2018 •
edited
Loading

@marcoabreu @zheng-da @azai91 @pengzhao-intel
We have found the root cause of this issue and here is the analysis:
The test failed at MKLDNNSum in place test case, and failed due to delay_alloc check failure in SetMKLMem(),

void NDArray::Chunk::SetMKLMem(const TShape &shape, int dtype) {
...
  if (shandle.dptr == nullptr) {
    CHECK(delay_alloc);         <<< check failure here
    CheckAndAlloc();
  }
...

it indicates the related NDArray's shandle is nullptr but delay_alloc is set to false, usually delay_alloc will only be set to false after shandle is allocated, so most probably there is race condition here (this issue is very hard to reproduce, only happens occasionally), and I tried to print delay_alloc 2 times before CHECK(delay_alloc) and it is found when issue happens delay_alloc is true at first but then become false for the second time! This confirms that delay_alloc must be set to false by another thread asynchronously and cause thread race condition (print will make the race condition easier to happen). By checking the test code it is found there may exist race condition for below 2 calls (commented lines)

TEST(MKLDNN_BASE, MKLDNNSum) {
 ...
    NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy");    <<< Copy will push operation to engine asynchronously which will alloc shandle for orig_arr.arr
    PrintVerifyMsg(orig_arr, in_arr);
    InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc());     <<< may alloc shandle for orig_arr.arr which cause conflict
    orig_arr.arr.CopyFrom(*input_mem);
...
}

The detailed function calls is as below:

NDArray::Copy()
  --> CopyFromTo(*this, ret)
    --> Engine::Get()->PushAsync(...,   
           CopyFromToImpl<cpu, cpu>(from, to, ctx, requested) ...)
              --> CopyFromToDnsImpl()
                  --> to.GetMKLDNNData()
                     --> ptr_->SetMKLMem()
                       --> CHECK(delay_alloc) 
                       -->  CheckAndAlloc()
                          --> alloc shandle and set delay_alloc to false
                         

InitMKLDNNArray()
  --> InitDefaultArray(arr, is_rand)
    --> arr->data()
      --> CheckAndAlloc()
         --> alloc shandle and set delay_alloc to false

InitMKLDNNArray() happens in test main thread while CopyFromToImpl() happens asynchronously in Engine thread, so there may exist thread race condition here that InitMKLDNNArray() set delay_alloc to false and cause the CHECK failed for CopyFromToImpl().

A simple fix is to add barrier to wait for Copy operation to finish before further operations to get rid of the race condition. The proposed change is as below:


diff --git a/tests/cpp/operator/mkldnn.cc b/tests/cpp/operator/mkldnn.cc
index 4f8f6f1..59bd3a5 100644
--- a/tests/cpp/operator/mkldnn.cc
+++ b/tests/cpp/operator/mkldnn.cc
@@ -1308,6 +1308,7 @@ TEST(MKLDNN_BASE, MKLDNNSum) {
     auto input_mem = in_arr.arr.GetMKLDNNData();
     auto input_mem2 = in_arr2.arr.GetMKLDNNData();
     NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy");
+    orig_arr.arr.WaitToRead();
     PrintVerifyMsg(orig_arr, in_arr);
     InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc());
     orig_arr.arr.CopyFrom(*input_mem);

@jinhuang415 jinhuang415 mentioned this issue Aug 8, 2018
Merged
5 tasks
@pengzhao-intel
Copy link
Contributor

In short, the CPP test case mixed the usage of MXNET OP and C/C++ function where the MXNET OP is executed with async mode and C function will run directly.
So, this will bypass the dependency engine and then cause the data race.

In concluding, this is NOT the bug in MKLDNN or dependency engine.
Just a not common implementation of the test case.

@marcoabreu
Copy link
Contributor Author

Sorry, I'm currently swamped with high priority tasks and I don't have time to review your task or assist with inqueries.

@anirudh2290
Copy link
Member

Fixed by #12080

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Disabled test Flaky Test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@anirudh2290 @lupesko @marcoabreu @pengzhao-intel @jinhuang415