Improved training speed of GHNs, extra results for CIFAR-10

[bknyaz]

Training times

Implementation of some steps in the decoder of GHNs is improved to speed up the training time of GHNs without altering their overall behavior. These improvements mainly affect the speed when a meta-batch size > 1 is used (see the tables below).

Speed is measured on NVIDIA Quadro RTX 6000 in terms of seconds per training iteration (averaged for the first 100 iterations).

CIFAR-10

Model	Current version	Our PR	Estimated total speed up for 300 epochs
MLP with meta-batch size bm = 1	0.21 sec/iter	0.13 sec/iter	0.5 days -> 0.3 days
MLP with meta-batch size bm = 8	6.35 sec/iter	0.89 sec/iter	15.5 days -> 2.2 days
GHN-2 with meta-batch size bm = 1	0.77 sec/iter	0.72 sec/iter	1.9 days -> 1.8 days
GHN-2 with meta-batch size bm = 8	7.74 sec/iter	1.99 sec/iter	18.9 days -> 4.9 days

ImageNet

Model	Current version	Our PR	Estimated total speed up for 300/150 epochs*
MLP with bm = 1	0.53 sec/iter	0.37 sec/iter	8.9 days -> 6.2 days
MLP with bm = 8 (4 GPUs)	1.78 sec/iter	1.36 sec/iter	14.9 days -> 11.4 days
GHN-2 with bm = 1	1.08 sec/iter	0.92 sec/iter	18.0 days -> 15.4 days
GHN-2 with bm = 8 (4 GPUs)	3.78 sec/iter	3.50 sec/iter	31.6 days -> 29.2 days

*To estimate the total training time, 300 epochs is used for bm=1 and 150 epochs is used for bm=8 (according to the paper).

When 4 GPUs and bm = 8 is used, the speed up is not significant, because each GPU receives only two architectures.

Evaluation of GHNs

To make sure that the evaluation results (classification accuracies of predicted parameters) reported in the paper are the same as in this PR, the GHNs were evaluated on selected architectures and the same results were obtained (see the table below).

Model	ResNet-50	ViT	Test Architecture (index in the test split)
GHN-2-CIFAR-10	58.6	11.4	77.1 (210)
GHN-2-ImageNet	5.3	4.4	48.3 (85)

Extra results on CIFAR-10

• Added file data/results_cifar10_extended.json containing training and evaluation curves (when training with SGD) for 1402 evaluation architectures of DeepNets-1M.
• The results of evaluation GHNs on validation and noisy images of CIFAR-10 are added.
• Jupyter notebook examples/read_cifar10_results.ipynb is added to read and visualize the training/evaluation curves.
  The image below is obtained using this notebook.

Other minor updates

• --amp flag was added that can be used to decrease GPU memory consumption and, in some cases, improve speed (this flag was used to measure speed on ImageNet with 4 GPUs)
• md5sum values of the DeepNets-1M files were added to make it easier to verify the dataset