Fine-tuning InceptionV3: High accuracy during training, low during predictions

[jksmither]

I installed Keras 2.1.6 and TensorFlow 1.7 and I'm using an example straight from Keras Documentation to fine-tune an InceptionV3 on a new set of 5 classes. My code is identical to the documentation; I freeze part of the network and train the rest. The problem is that during training my accuracy is high but during predictions is extremely low.

The first thing that came in my mind was overfitting so I investigated but there are no signs of that. Then I tried using the same dataset for training and validation and to my surprise I got completely different accuracy at the end of fit() and after calling validate(). Again I'm using the same dataset and at the end of training I get 93% accuracy and immediately after validation 25%!!!

Any help will be highly appreciated!

[SpecKROELLchen]

I think this is an issue that other people, especially me, encounter too.
I use the same version of keras and tensorflow.
It was stated that it has to do with the batch normalization as you can see here #9965 and in my post #10040.
It did not help me, but maybe you will find out a work around.
Also, try another net like vgg16, it was stated that these problems don't arise there.
But when i tested vgg16 even the training accuracy got stuck at a random value of 65% and the net did not train properly, while for resnet50 and inceptionv3 i encountered exactly your problem that even testing on training data gave a low accuracy result.

[jksmither]

@SpecKROELLchen Thanks for providing the references! I confirm that using architectures without batch normalization like VGG "solves" the problem.

I also found the issue #9214 which describes a similar situation. Reading more into #9965, I understand that there is no consensus among maintainers on whether this is a bug or a feature. One thing I don't get, is why on earth would you ever want to standardize differently the data in Train and Test mode when the layers are not-trainable. This does not make sense to me at all. I am using code directly taken from documentation, this ought to work.

To work around the problem I tried K.set_learning_phase(1). I do this at the very beginning before I even load the model or start training. After training is completed, the accuracy agrees both in training and validation. But that's not a real solution because I can't deploy a model in training mode. The way I understand it, with learning phase set to 1 the predictions for a given record will depend on the other records of the batch.

[SpecKROELLchen]

@jksmither Honestly, i also don`t get it.
Thx for trying it out with VGG, because i ran
into another problem while trying out vgg, which makes me feel that i am doing something wrong
in addition to the BN problem..

Can you or anyone try to reproduce it with this code or find what i did wrong?
I have posted it here #10014

[lovekesh-thakur]

I am trying to fine tune using Inception V3 on a dataset containing 40K images in two classes (balanced).
For stage 1, i froze all base layers and just trained 2 fc layers (1024 units and 2 respectively). My training accuracy went 95 in 20 epochs while my validation accuracy is stuck at 49% to 53%.
After going through threads, i switched my base model to VGG16 and my validation accuracy is at 88%.
I will update my post when training is over.