[Question] In paper Table 6, why variant (d) is better than variant (c)?

[pkumc]

@LinB203 Really appreciate your work! I have some questions.

Question 1

In table 6, if I understand correctly, Variant (c) and variant (d) are both dense model. Variant (c) use LV+Hb data to train all parameters. Variant (d) use Hb data to train all parameters, but use LV data to only train FFN parameters(other parameters are frozen). Why variant (d) can get better result than variant (c)?

Question 2

In table 5, why 'expert_num=1 & topK=1 (the first line of table b)' is better than 'expert_num=4 & topK=1 (the first line of table c)'? It indicates MoE can not beat the dense counterpart with the same activated parameters. Only when topK=2, the activated parameters is double, MoE can win.
And according to the paper's description, the first line of table b should be same as variant (d) of table 6, but the results show it is same as variant (c) of table 6.