research-article

Robust Training of Graph Neural Networks via Noise Governance

Authors:

Jian WuAuthors Info & Claims

WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

Pages 607 - 615

https://doi.org/10.1145/3539597.3570369

Published: 27 February 2023 Publication History

Abstract

Graph Neural Networks (GNNs) have become widely-used models for semi-supervised learning. However, the robustness of GNNs in the presence of label noise remains a largely under-explored problem. In this paper, we consider an important yet challenging scenario where labels on nodes of graphs are not only noisy but also scarce. In this scenario, the performance of GNNs is prone to degrade due to label noise propagation and insufficient learning. To address these issues, we propose a novel RTGNN (<u>R</u>obust <u>T</u>raining of <u>G</u>raph <u>N</u>eural <u>N</u>etworks via Noise Governance) framework that achieves better robustness by learning to explicitly govern label noise. More specifically, we introduce self-reinforcement and consistency regularization as supplemental supervision. The self-reinforcement supervision is inspired by the memorization effects of deep neural networks and aims to correct noisy labels. Further, the consistency regularization prevents GNNs from overfitting to noisy labels via mimicry loss in both the inter-view and intra-view perspectives. To leverage such supervisions, we divide labels into clean and noisy types, rectify inaccurate labels, and further generate pseudo-labels on unlabeled nodes. Supervision for nodes with different types of labels is then chosen adaptively. This enables sufficient learning from clean labels while limiting the impact of noisy ones. We conduct extensive experiments to evaluate the effectiveness of our RTGNN framework, and the results validate its consistent superior performance over state-of-the-art methods with two types of label noises and various noise rates.

Supplementary Material

MP4 File (WSDM23-fp0031.mp4)

Presentation video.

Download
25.05 MB

References

[1]

Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018. Graph based semi-supervised learning with convolution neural networks to classify crisis related tweets. In 12th International AAAI Conference on Web and Social Media.

[2]

Devansh Arpit, Stanisław Jastrzke bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at memorization in deep networks. In International Conference on Machine Learning. PMLR, 233--242.

[3]

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral Networks and Locally Connected Networks on Graphs. In 2nd International Conference on Learning Representations, ICLR.

[4]

Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2019. Deep iterative and adaptive learning for graph neural networks. arXiv preprint arXiv:1912.07832 (2019).

[5]

Luca Cosmo, Anees Kazi, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. 2020. Latent-graph learning for disease prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 643--653.

Digital Library

[6]

Enyan Dai, Charu Aggarwal, and Suhang Wang. 2021. NRGNN: Learning a Label Noise Resistant Graph Neural Network on Sparsely and Noisily Labeled Graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 227--236.

Digital Library

[7]

Enyan Dai, Wei Jin, Hui Liu, and Suhang Wang. 2022. Towards Robust Graph Neural Networks for Noisy Graphs with Sparse Labels. In WSDM'22: The 15th ACM International Conference on Web Search and Data Mining.

[8]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in Neural Information Processing Systems, Vol. 29 (2016).

[9]

Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. 2021. SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks. Advances in Neural Information Processing Systems, Vol. 34 (2021).

[10]

Aritra Ghosh, Himanshu Kumar, and PS Sastry. 2017. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.

[11]

Jacob Goldberger and Ehud Ben-Reuven. 2017. Training deep neural-networks using a noise adaptation layer. In 5th International Conference on Learning Representations (ICLR) 2017.

[12]

Jonathan Halcrow, Alexandru Mosoi, Sam Ruth, and Bryan Perozzi. 2020. Grale: Designing networks for graph learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2523--2532.

Digital Library

[13]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, Vol. 30 (2017).

[14]

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems, Vol. 31 (2018).

[15]

Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In 4th International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.).

[16]

Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. 2019. O2Uu-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3326--3334.

[17]

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. PMLR, 2304--2313.

[18]

Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. 2020. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 66--74.

Digital Library

[19]

Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. 2022. Differentiable graph module (DGM) for graph convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[20]

Thomas N Kipf and Max Welling. 2016a. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).

[21]

Thomas N Kipf and Max Welling. 2016b. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).

[22]

Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning Sparse Neural Networks through $L_0$ Regularization. In 6th International Conference on Learning Representations (ICLR).

[23]

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. 2020. Normalized loss functions for deep learning with noisy labels. In International Conference on Machine Learning. PMLR, 6543--6553.

[24]

Eran Malach and Shai Shalev-Shwartz. 2017. Decoupling "when to update" from "how to update". Advances in Neural Information Processing Systems, Vol. 30 (2017).

[25]

Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a Feather: Homophily in Social Networks. Annual Review of Sociology, Vol. 27, 1 (2001), 415--444.

[26]

Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2020. SELF: Learning to Filter Noisy Labels with Self-Ensembling. In 8th International Conference on Learning Representations (ICLR).

[27]

Hoang NT, Choong Jun Jin, and Tsuyoshi Murata. 2019. Learning graph neural networks with noisy labels. arXiv preprint arXiv:1905.01591 (2019).

[28]

Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. 2018. Adversarially regularized graph autoencoder for graph embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2609--2615.

[29]

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. 2017. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1944--1952.

[30]

Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. 2020. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, Vol. 33 (2020), 17044--17056.

[31]

Petar Velivc ković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).

[32]

Daixin Wang, Jianbin Lin, Peng Cui, Quanhui Jia, Zhen Wang, Yanming Fang, Quan Yu, Jun Zhou, Shuang Yang, and Yuan Qi. 2019a. A Semi-supervised Graph Attentive Network for Financial Fraud Detection. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 598--607.

[33]

Deng-Bao Wang, Yong Wen, Lujia Pan, and Min-Ling Zhang. 2021. Learning from noisy labels with complementary loss functions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 10111--10119.

[34]

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. 2019b. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 322--330.

[35]

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. 2020. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13726--13735.

[36]

Jun Wu, Jingrui He, and Jiejun Xu. 2019. DEMO-Net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 406--415.

Digital Library

[37]

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. 2021. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00445 (2021).

[38]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).

[39]

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. 2016. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning. PMLR, 40--48.

[40]

Qing Yu, Atsushi Hashimoto, and Yoshitaka Ushiku. 2021. Divergence Optimization for Noisy Universal Domain Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2515--2524.

[41]

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. 2019. How does disagreement help generalization against label corruption?. In International Conference on Machine Learning. PMLR, 7164--7173.

[42]

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations (ICLR).

[43]

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4320--4328.

[44]

Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems, Vol. 31 (2018).

[45]

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. 2021. Data augmentation for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11015--11023.

[46]

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11106--11115.

Cited By

Li DLi XGan ZLi QQu BWang J(2025)Rethinking the impact of noisy labels in graph classification: A utility and privacy perspectiveNeural Networks10.1016/j.neunet.2024.106919182(106919)Online publication date: Feb-2025
https://doi.org/10.1016/j.neunet.2024.106919
Yuan XTian YZhang CYe YChawla NZhang CBaeza-Yates RBonchi F(2024)Graph Cross Supervised Learning via Generalized KnowledgeProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671830(4083-4094)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671830
Cheng YShan CShen YLi XLuo SLi DBaeza-Yates RBonchi F(2024)Resurrecting Label Propagation for Graphs with Heterophily and Label NoiseProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671774(433-444)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671774
Show More Cited By

Index Terms

Robust Training of Graph Neural Networks via Noise Governance
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Semi-supervised learning settings

Recommendations

Unified Robust Training for Graph Neural Networks Against Label Noise
Advances in Knowledge Discovery and Data Mining
Abstract
Graph neural networks (GNNs) have achieved state-of-the-art performance for node classification on graphs. The vast majority of existing works assume that genuine node labels are always provided for training. However, there has been very little ...
Analysis of label noise in graph-based semi-supervised learning
SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

In machine learning, one must acquire labels to help supervise a model that will be able to generalize to unseen data. However, the labeling process can be tedious, long, costly, and error-prone. It is often the case that most of our data is unlabeled. ...
Transductive Multilabel Learning via Label Set Propagation

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

February 2023

1345 pages

ISBN:9781450394079

DOI:10.1145/3539597

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Hady Lauw
Singapore Management University
,
Program Chairs:
Luo Si
Salesforce
,
Evimaria Terzi
Boston University
,
Panayiotis Tsaparas
University of Ioannina

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

WSDM '23

Sponsor:

WSDM '23: The Sixteenth ACM International Conference on Web Search and Data Mining

February 27 - March 3, 2023

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
323
Total Downloads

Downloads (Last 12 months)143
Downloads (Last 6 weeks)15

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li DLi XGan ZLi QQu BWang J(2025)Rethinking the impact of noisy labels in graph classification: A utility and privacy perspectiveNeural Networks10.1016/j.neunet.2024.106919182(106919)Online publication date: Feb-2025
https://doi.org/10.1016/j.neunet.2024.106919
Yuan XTian YZhang CYe YChawla NZhang CBaeza-Yates RBonchi F(2024)Graph Cross Supervised Learning via Generalized KnowledgeProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671830(4083-4094)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671830
Cheng YShan CShen YLi XLuo SLi DBaeza-Yates RBonchi F(2024)Resurrecting Label Propagation for Graphs with Heterophily and Label NoiseProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671774(433-444)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671774
Zhang JJiang XTian NWu M(2024)Label noise correction for crowdsourcing using dynamic resamplingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108439133(108439)Online publication date: Jul-2024
https://doi.org/10.1016/j.engappai.2024.108439
Liu XMeng SLi QQi LXu XDou WZhang XFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)SMEF: Social-aware Multi-dimensional Edge Features-based Graph Representation Learning for RecommendationProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615063(1566-1575)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615063
Yuan JLuo XQin YMao ZJu WZhang MEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)ALEX: Towards Effective Graph Transfer Learning with Noisy LabelsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612026(3647-3656)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612026

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents