Skip to content

[NeurIPS 2022] Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again by Ajay Jaiswal*, Peihao Wang*, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang

Notifications You must be signed in to change notification settings

VITA-Group/GradientGCN

Repository files navigation

Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again

License: MIT

https://arxiv.org/abs/2210.08122

Abstract

Despite the enormous success of Graph Convolutional Networks (GCNs) in mod- elling graph-structured data, most of the current GCNs are shallow due to the notoriously challenging problems of over-smoothening and information squashing along with conventional difficulty caused by vanishing gradients and over-fitting. Previous works have been primarily focused on the study of over-smoothening and over-squashing phenomenon in training deep GCNs. Surprisingly, in comparison with CNNs/RNNs, very limited attention has been given towards understanding how healthy gradient flow can benefit the trainability of deep GCNs. In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the- art (SOTA) level performance from vanilla-GCNs [1]. Next, we argue that blindly adopting the Glorot initialization for GCNs is not optimal, and derive a topology- aware isometric initialization scheme for vanilla-GCNs based on the principles of isometry. Additionally, contrary to ad-hoc addition of skip-connections, we propose to use gradient-guided dynamic rewiring of vanilla-GCNs with skip- connections. Our dynamic rewiring method uses the gradient flow within each layer during training to introduce skip-connections on-demand basis. We provide extensive empirical evidence across multiple datasets that our methods improves gradient flow in deep vanilla-GCNs and significantly boost their performance to comfortably compete and outperform many fancy state-of-the-art methods.

image

Benefits of our proposed techniques

image

image

image

image

If you find our work helpful in your research, please cite our paper

Citation

If you find our code implementation helpful for your own resarch or work, please cite our paper.

@inproceedings{Jaiswal22GradientGCN,
  title={Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again},
  author={Ajay Jaiswal, Peihao Wang, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang},
  booktitle={NeurIPS 2022},
  year={2022}
}

About

[NeurIPS 2022] Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again by Ajay Jaiswal*, Peihao Wang*, Tianlong Chen, Justin F Rousseau, Ying Ding, Zhangyang Wang

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages