research-article

Public Access

Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective

Authors:

Charu Aggarwal,

Jiliang TangAuthors Info & Claims

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 709 - 719

https://doi.org/10.1145/3534678.3539445

Published: 14 August 2022 Publication History

Abstract

Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.

References

[1]

Babajide O Ayinde, Tamer Inanc, and Jacek M Zurada. 2019. On correlation of features extracted by deep neural networks. In IJCNN.

[2]

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. 2018. Mine: mutual information neural estimation. ArXiv preprint (2018).

[3]

Anthony J Bell and Terrence J Sejnowski. 1995. An information-maximization approach to blind separation and blind deconvolution. In Neural computation.

[4]

Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing.

[5]

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral Networks and Locally Connected Networks on Graphs. In ICLR.

[6]

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020 a. Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View. In AAAI.

[7]

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020 b. Simple and Deep Graph Convolutional Networks. In ICML.

[8]

Michael Cogswell, Faruk Ahmed, Ross B. Girshick, Larry Zitnick, and Dhruv Batra. 2016. Reducing Overfitting in Deep Networks by Decorrelating Representations. In ICLR.

[9]

Michaë l Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS.

[10]

Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. 2022. Data augmentation for deep graph learning: A survey. arXiv preprint arXiv:2202.08235 (2022).

[11]

David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gó mez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In NeurIPS.

[12]

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph Neural Networks for Social Recommendation. In WWW.

[13]

Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. ArXiv preprint (2019).

[14]

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural Message Passing for Quantum Chemistry. In ICML.

Digital Library

[15]

Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. 2021. Few-shot graph learning for molecular property prediction. In Proceedings of the Web Conference 2021.

Digital Library

[16]

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NeurIPS.

[17]

Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. ArXiv preprint (2015).

[18]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML.

Digital Library

[19]

Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. 2022. Automated Self-Supervised Learning for Graphs. In ICLR.

[20]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.

[21]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.

[22]

Guohao Li, Matthias Mü ller, Bernard Ghanem, and Vladlen Koltun. 2021. Training Graph Neural Networks with 1000 Layers. In ICML.

[23]

Guohao Li, Matthias Mü ller, Ali K. Thabet, and Bernard Ghanem. 2019. DeepGCNs: Can GCNs Go As Deep As CNNs?. In ICCV.

[24]

Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. In AAAI.

[25]

Meng Liu, Hongyang Gao, and Shuiwang Ji. 2020. Towards Deeper Graph Neural Networks. In KDD.

[26]

Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. 2021. Elastic Graph Neural Networks. In ICML.

[27]

Diego Marcheggiani, Jasmijn Bastings, and Ivan Titov. 2018. Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks. In NAACL.

[28]

Kenta Oono and Taiji Suzuki. 2020. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. In ICLR.

[29]

Liam Paninski. 2003. Estimation of entropy and mutual information. Neural computation 6 (2003).

[30]

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2020. Geom-GCN: Geometric Graph Convolutional Networks. In ICLR.

[31]

Al Mamunur Rashid, George Karypis, and John Riedl. 2008. Learning preferences of new users in recommender systems: an information theoretic approach. Acm Sigkdd Explorations Newsletter 2 (2008).

Digital Library

[32]

Pau Rodríguez, Jordi Gonzàlez, Guillem Cucurull, Josep M. Gonfaus, and F. Xavier Roca. 2017. Regularizing CNNs with Locally Constrained Decorrelations. In ICLR.

[33]

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In ICLR.

[34]

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine 3 (2008).

[35]

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. ArXiv preprint (2018).

[36]

David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE signal processing magazine 3 (2013).

[37]

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR.

[38]

Yu Wang, Wei Jin, and Tyler Derr. 2022 a. Graph Neural Networks: Self-supervised Learning. In Graph Neural Networks: Foundations, Frontiers, and Applications. Springer, 391--420.

[39]

Yu Wang, Yuying Zhao, Yushun Dong, Huiyuan Chen, Jundong Li, and Tyler Derr. 2022 b. Improving Fairness in Graph Neural Networks via Mitigating Sensitive Attribute Leakage. In KDD.

[40]

Hongzhi Wen, Jiayuan Ding, Wei Jin, Yiqi Wang, Yuying Xie, and Jiliang Tang. 2022. Graph Neural Networks for Multimodal Single-Cell Data Integration. In KDD.

[41]

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. 2019. A comprehensive survey on graph neural networks. ArXiv preprint (2019).

[42]

Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. In AAAI.

[43]

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. 2021. Graph Contrastive Learning Automated. In ICML.

[44]

Lingxiao Zhao and Leman Akoglu. 2020. PairNorm: Tackling Oversmoothing in GNNs. In ICLR.

[45]

Tong Zhao, Gang Liu, Stephan Günnemann, and Meng Jiang. 2022. Graph Data Augmentation for Graph Machine Learning: A Survey. arXiv preprint arXiv:2202.08871 (2022).

[46]

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. 2021. Data augmentation for graph neural networks. In AAAI.

[47]

Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2018. Graph neural networks: A review of methods and applications. ArXiv preprint (2018).

[48]

Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. 2020. Towards Deeper Graph Neural Networks with Differentiable Group Normalization. In NeurIPS.

[49]

Marinka Zitnik, Monica Agrawal, and Jure Leskovec. 2018. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics 13 (2018).

Cited By

Gao HLi NXie Y(2024)Hidden-SAGE: For the Inference of Complex Autonomous System Business Relationships Involving Hidden LinksElectronics10.3390/electronics1309161713:9(1617)Online publication date: 23-Apr-2024
https://doi.org/10.3390/electronics13091617
陈丹(2024)Algorithm Research of Attention Mechanism in Graph Neural Network ModelModeling and Simulation10.12677/MOS.2024.13102213:01(225-238)Online publication date: 2024
https://doi.org/10.12677/MOS.2024.131022
Yao KLi W(2024)Asymmetric Learning for Graph Neural Network based Link PredictionACM Transactions on Knowledge Discovery from Data10.1145/364034718:5(1-18)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1145/3640347
Show More Cited By

Index Terms

Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Semi-supervised learning settings
    2. Machine learning approaches
      1. Neural networks

Recommendations

Toward the analysis of graph neural networks
ICSE-NIER '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

Graph Neural Networks (GNNs) have recently emerged as an effective framework for representing and analyzing graph-structured data. GNNs have been applied to many real-world problems such as knowledge graph analysis, social networks recommendation, and ...
Symmetric Rectified Linear Units for Fully Connected Deep Models
Knowledge Science, Engineering and Management
Abstract
Rectified Linear Units (ReLU) is one of the key aspects for the success of Deep Learning models. It has been shown that deep networks can be trained efficiently using ReLU without pre-training. In this paper, we compare and analyze various kinds ...
On the Power of Graph Neural Networks and Feature Augmentation Strategies to Classify Social Networks
Intelligent Information and Database Systems
Abstract
This paper studies four Graph Neural Network architectures (GNNs) for a graph classification task on a synthetic dataset created using classic generative models of Network Science. Since the synthetic networks do not contain (node or edge) ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2022

5033 pages

ISBN:9781450393850

DOI:10.1145/3534678

General Chairs:
Aidong Zhang
University of Virginia
,
Huzefa Rangwala
Amazon/George Mason University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

KDD '22

Sponsor:

KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2022

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
766
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)34

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gao HLi NXie Y(2024)Hidden-SAGE: For the Inference of Complex Autonomous System Business Relationships Involving Hidden LinksElectronics10.3390/electronics1309161713:9(1617)Online publication date: 23-Apr-2024
https://doi.org/10.3390/electronics13091617
陈丹(2024)Algorithm Research of Attention Mechanism in Graph Neural Network ModelModeling and Simulation10.12677/MOS.2024.13102213:01(225-238)Online publication date: 2024
https://doi.org/10.12677/MOS.2024.131022
Yao KLi W(2024)Asymmetric Learning for Graph Neural Network based Link PredictionACM Transactions on Knowledge Discovery from Data10.1145/364034718:5(1-18)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1145/3640347
Wu WWang CShen DQin CChen LXiong HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)AFDGCF: Adaptive Feature De-correlation Graph Collaborative Filtering for RecommendationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657724(1242-1252)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657724
Dong CLi DLiu J(2024)Glass Transition Temperature Prediction of Polymers via Graph Reinforcement LearningLangmuir10.1021/acs.langmuir.4c0190640:35(18568-18580)Online publication date: 21-Aug-2024
https://doi.org/10.1021/acs.langmuir.4c01906
Li ZJia MWei ZWang J(2024)Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularizationNeural Networks10.1016/j.neunet.2024.106704180(106704)Online publication date: Dec-2024
https://doi.org/10.1016/j.neunet.2024.106704
Qiang YDong XLiu XYang YFang YDou J(2024)Conv-RGNN: An efficient Convolutional Residual Graph Neural Network for ECG classificationComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2024.108406(108406)Online publication date: Sep-2024
https://doi.org/10.1016/j.cmpb.2024.108406
Zhou DYang HXiong BMa YKharlamov E(2024)Alleviating Over-Smoothing via Aggregation over Compact ManifoldsAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2253-2_31(390-404)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1007/978-981-97-2253-2_31
Yang MFeng WShen YHooi BKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Towards better graph representation learning with parameterized decomposition & filteringProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3620044(39234-39251)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3620044
Lee SBu FYoo JShin KKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Towards deep attention in graph neural networksProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619183(18774-18795)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619183
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten