The results of Soft Logits fluctuate quite a lot

[zhongshaoyy]

Hi,
when train the student network using soft logits method and running the code:
python3 train_w_distill2.py --Distillation=Soft_logits --train_dir=soft_logits --main_scope=Student_w_Soft_logits --teacher=ResNet32.mat ,I find that the results changes a lot . After training five times, I get 71.48, 72.25, 72.34, 71.73, 72.46 (Best Accuracy)for the student network. Is this normal?

[sseung0703]

If you get an average of them, it is similar to my experimental results.
My results are
Last : [71.63, 71.80, 71.87, 71.66, 71.98]
Best : [71.89, 72.41, 71.97, 72.05, 72.10]
We rarely consider the best accuracy. I presented this just for additional information.
Focus on the last. In our case, they are not so fluctuated.

[zhongshaoyy]

OK， Thank you for the reply. I got another question. We can see that the soft logits method doesn't get a pretty result( Best accuacy for student network from 71.76 to 71.79). Have you try other settings,?such as T=10 or higher coefficent for the KL loss

[sseung0703]

No, I didn't try so many times for each method. There are so many configurations for knowledge distillation, such as hyper-parameters, feature sensing points, and so on, which is hard to find the optimal one. If you want, you can find a better configuration for them.