Problem converting ResNeXt-50 MXNet -> Keras

[qubvel]

Platform (like ubuntu 16.04/win10):
ubuntu 16.04

Python version:
3.6

Source framework with version (like Tensorflow 1.4.1 with GPU):
mxnet 1.2.0 (cpu)

Destination framework with version (like CNTK 2.3 with GPU):
keras (cpu) 2.1.5 (tensorflow 1.7)

Pre-trained model path (webpath or webdisk path):
imagenet1k-resnext-50 (mxnet networks zoo)

Running scripts:
I follow the issue #58 but got the errors while converting the model:

$ python3 -m mmdnn.conversion._script.convertToIR -f mxnet -n ./resnext-50-symbol.json -w ./resnext-50-0000.params -d mxnet_resnext50 --inputShape 3 224 224
$ python3 -m mmdnn.conversion._script.IRToCode -f keras --IRModelPath mxnet_resnext50.pb --dstModelPath converted_resnext50.py --IRWeightPath mxnet_resnext50.npy
$ python3 -m mmdnn.conversion.examples.keras.imagenet_test -p imagenet1k-resnext-50 -s mxnet -n converted_resnext50 -w mxnet_resnext50.npy -i ./seagull.jpg

1. Local variable 'labels' referenced before assignment here - missed indentation level

2. Test fails

File "/media/D/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/examples/imagenet_test.py", line 329, in test_truth 
assert np.isclose(this_truth[index][1], i[1], atol = 1e-6)
AssertionError 

I converted model as it is using:
mmconvert -sf mxnet -in resnext-50-symbol.json -iw resnext-50-0000.params -df keras -om resnext50.h5 --inputShape 3 224 224

And print sum of outputs of first model layers

Script

from keras.models import load_model
from keras.models import Model
import numpy as np

model = load_model('../resnext50/resnext50.h5')

def get_output_at_layer(model, layer, x):
    input = model.input
    output = model.layers[layer].output
    new_model = Model(input, output)
    return new_model.predict(x)

x = np.random.random((1,224,224,3)) * 255

for i in range(50):
    y = get_output_at_layer(model, i, x)
    print(model.layers[i].name, np.sum(y))

Output

data 19219074.0
bn_data 94388.93
zero_padding2d_1 94388.92
conv0 49383.504
bn0 443316.25
relu0 644856.3
zero_padding2d_2 644856.4
pooling0 253928.1
stage1_unit1_conv1 -197160.0
stage1_unit1_bn1 -81262.35
stage1_unit1_relu1 56259.15
zero_padding2d_3 56259.15
lambda_1 2100.377
lambda_2 2100.377
lambda_3 2100.377
lambda_4 2100.377
lambda_5 2100.377
lambda_6 2100.377
lambda_7 2100.377
lambda_8 2100.377
lambda_9 2100.377
lambda_10 2100.377
lambda_11 2100.377
lambda_12 2100.377
lambda_13 2100.377
lambda_14 2100.377
lambda_15 2100.377
lambda_16 2100.377
lambda_17 2100.377
lambda_18 2100.377
lambda_19 2100.377
lambda_20 2100.377
lambda_21 2100.377
lambda_22 2100.377
lambda_23 2100.377
lambda_24 2100.377
lambda_25 2100.377
lambda_26 2100.377
lambda_27 2100.377
lambda_28 2100.377
lambda_29 2100.377
lambda_30 2100.377
lambda_31 2100.377
lambda_32 2100.377

Thats seems to be strange to have the same output from all Lambda layers, where previous layer has different values

Script

i = 11
y = get_output_at_layer(model, i, x)
print(model.layers[i].name, np.sum(y))

for i in range(32):
    print('slice {}:{} \t'.format(i*4, (i+1)*4), np.sum(y[..., i*4:(i+1)*4]))

Output

zero_padding2d_3 56259.15
slice 0:4 	    3857.7256
slice 4:8 	    1897.7053
slice 8:12 	 46.061497
slice 12:16 	 4812.0205
slice 16:20 	 2283.8503
slice 20:24 	 54.295998
slice 24:28 	 20.24092
slice 28:32 	 1976.638
slice 32:36 	 662.6427
slice 36:40 	 2185.7913
slice 40:44 	 333.3028
slice 44:48 	 3391.289
slice 48:52 	 4329.0356
slice 52:56 	 431.72855
slice 56:60 	 0.0
slice 60:64 	 663.5157
slice 64:68 	 1484.2971
slice 68:72 	 753.3141
slice 72:76 	 594.09753
slice 76:80 	 4688.849
slice 80:84 	 353.21588
slice 84:88 	 0.0
slice 88:92 	 457.56436
slice 92:96 	 3652.3367
slice 96:100 	 818.3705
slice 100:104 	 5672.6484
slice 104:108 	 0.0
slice 108:112 	 1170.7343
slice 112:116 	 5348.7695
slice 116:120 	 2218.7273
slice 120:124 	 0.0
slice 124:128 	 2100.377

I dont know why, but all Lambda layers have output as the last 4 feature maps of previous layer.

[namizzz]

Hi @qubvel

 python3 -m mmdnn.conversion.examples.keras.imagenet_test -p imagenet1k-resnext-50 -s mxnet -n converted_resnext50 -w mxnet_resnext50.npy -i ./seagull.jpg

It's a script for test the accuracy of the conversion, not a must. Error "Local variable 'labels' referenced before assignment" is because that there is no file 'mmdnn/conversion/examples/data/imagenet_1000.txt'. You need to check the newest code.

Lambda layer is for group convolution, you can check the converted code file. I test the accuracy of the conversion from resnext50 mxnet to keras. The error is e-6. It's right.

[qubvel]

Hi @namizzz
Thanks for comment, could you please share converted model?

I just want to clarify following:

1. Should not this piece of code(lines 310-316) corrected as follows to avoid the Error Local variable 'labels' referenced before assignment (added indentation for for cycle), so in case i don`t have the file I will not get the error:

now:

                import os
                if os.path.exists(self.args.label):
                    with open(self.args.label, 'r') as f:
                        labels = [l.rstrip() for l in f]

                for i in top_indices:
                    print (labels[i - offset], i, predict[i])

corrected:

                import os
                if os.path.exists(self.args.label):
                    with open(self.args.label, 'r') as f:
                        labels = [l.rstrip() for l in f]

                    for i in top_indices:
                        print (labels[i - offset], i, predict[i])

2. I convert the model even without testing conversion error, but still have incorrect results.
   My model (link) has result:
   [(380, 0.48661867), (630, 0.32678068), (257, 0.065482974), (392, 0.045178358), (463, 0.029203746)]
   correct result:
   [(396, 0.7104751), (398, 0.122665755), (438, 0.06391319), (440, 0.029796895), (417, 0.019492012)]

converting script:

$ mmdownload -f mxnet -n imagenet1k-resnext-50 -o ./
$ mmconvert -sf mxnet -in resnext-50-symbol.json -iw resnext-50-0000.params -df keras -om resnext50.h5 --inputShape 3 224 224

P.S. I have the same problem with ResNeXt101, however ResNet models coverts correctly.
P.P.S. Tested on other instance with tensorflow (1.7/1.5) and with GPU support with the same result

[kitstar]

Hi @qubvel,

Fixed the bug you mentioned. Thanks very much.

Reproduce the conversion with newest MMdnn(0.2.2) (the bug is not merged in newest pip package :( )

$ mmdownload -f mxnet -n imagenet1k-resnext-50 -i mmdnn/conversion/examples/data/seagull.jpg
(396, 0.9810116)
(398, 0.008996904)
(409, 0.0026355518)
(438, 0.0025629073)
(440, 0.0015916944)

$ mmtoir -f mxnet -n ./resnext-50-symbol.json -w ./resnext-50-0000.params -d mxnet_resnext50 --inputShape 3 224 224
IR network structure is saved as [mxnet_resnext50.json].
IR network structure is saved as [mxnet_resnext50.pb].
IR weights are saved as [mxnet_resnext50.npy].

$ mmtocode -f keras -in mxnet_resnext50.pb -iw mxnet_resnext50.npy -d converted_resnext50.py
Parse file [mxnet_resnext50.pb] with binary format successfully.
Target network code snippet is saved as [converted_resnext50.py].

$ python3 -m mmdnn.conversion.examples.keras.imagenet_test -p imagenet1k-resnext-50 -s mxnet -n converted_resnext50 -w mxnet_resnext50.npy -i mmdnn/conversion/examples/data/seagull.jpg
(396, 0.9810091)
(398, 0.008996932)
(409, 0.0026355372)
(438, 0.0025629275)
(440, 0.0015917055)

slightly different from MXNet result.

But after saving to keras .h5 file and load it again, the prediction result is same with yours. The IR weights file is 145MB, but the .h5 file is only 99M. No idea about this problem, maybe cause by Lambda layer. We will try to figure it out.

Thanks for reporting the problem again!

[qubvel]

Hi @kitstar
Thanks for the answer!
I checked it works correctly with loading weights from the *.npy and saving them to *.h5 just as weights (not as the whole model).
Thanks again, will wait for further updates with Lambda layer!