research-article

Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks

Authors:

Jie TangAuthors Info & Claims

KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Pages 1150 - 1160

https://doi.org/10.1145/3447548.3467350

Published: 14 August 2021 Publication History

Abstract

Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understanding of their advancements. In this work, we present a systematical reproduction of 12 recent HGNNs by using their official codes, datasets, settings, and hyperparameters, revealing surprising findings about the progress of HGNNs. We find that the simple homogeneous GNNs, e.g., GCN and GAT, are largely underestimated due to improper settings. GAT with proper inputs can generally match or outperform all existing HGNNs across various scenarios. To facilitate robust and reproducible HGNN research, we construct the Heterogeneous Graph Benchmark (HGB), consisting of 11 diverse datasets with three tasks. HGB standardizes the process of heterogeneous graph data splits, feature processing, and performance evaluation. Finally, we introduce a simple but very strong baseline Simple-HGN-which significantly outperforms all previous models on HGB-to accelerate the advancement of HGNNs in the future.

Supplementary Material

MP4 File (kdd21.mp4)

Presentation video of KDD'21 paper "Are we really making much progress? Revisiting, benchmarking and refining heterogeneous graph neural networks".

Download
413.90 MB

References

[1]

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML'19. PMLR, 21--29.

[2]

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et almbox. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018).

[3]

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD'08 . 1247--1250.

Digital Library

[4]

Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2011. Second workshop on information heterogeneity and fusion in recommender systems (HetRec2011). In RecSys'11. 387--388.

Digital Library

[5]

Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019. Representation learning for attributed multiplex heterogeneous network. In KDD'19 . 1358--1368.

Digital Library

[6]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).

[7]

Ming Ding, Jie Tang, and Jie Zhang. 2018. Semi-supervised learning on graphs with generative adversarial nets. In CIKM'18 . 913--922.

Digital Library

[8]

Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable Representation Learning for Heterogeneous Networks. In KDD '17. ACM, 135--144.

Digital Library

[9]

Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li. 2019 b. Metapath-guided heterogeneous graph neural network for intent recommendation. In KDD'19 . 2478--2486.

Digital Library

[10]

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019 a. Graph neural networks for social recommendation. In WWW'19. 417--426.

Digital Library

[11]

Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. 2020. Graph Random Neural Networks for Semi-Supervised Learning on Graphs. NeurIPS, Vol. 33 (2020).

[12]

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: metapath aggregated graph neural network for heterogeneous graph embedding. In WWW'20 .

Digital Library

[13]

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In ICML'17. PMLR, 1263--1272.

[14]

William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. arXiv preprint arXiv:1706.02216 (2017).

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR'16 .

[16]

Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. 2020. RealFormer: Transformer Likes Residual Attention. arXiv preprint arXiv:2012.11747 (2020).

[17]

Huiting Hong, Hantao Guo, Yucheng Lin, Xiaoqing Yang, Zang Li, and Jieping Ye. 2020. An attention-based graph neural network for heterogeneous structural learning. In AAAI'20, Vol. 34. 4132--4139.

[18]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020 c. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 (2020).

[19]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020 b. GPT-GNN: Generative Pre-Training of Graph Neural Networks. In KDD .

[20]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020 a. Heterogeneous graph transformer. In WWW'20. 2704--2710.

Digital Library

[21]

Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR'17 .

[22]

Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. 2020. Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739 (2020).

[23]

Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI'18, Vol. 32.

[24]

Danyang Liu, Jianxun Lian, Shiyin Wang, Ying Qiao, Jiun-Hung Chen, Guangzhong Sun, and Xing Xie. 2020. KRED: Knowledge-Aware Document Representation for News Recommendations. In RecSys'20 . 200--209.

[25]

Xichuan Niu, Bofang Li, Chenliang Li, Rong Xiao, Haochuan Sun, Hongbo Deng, and Zhenzhong Chen. 2020. A Dual Heterogeneous Graph Attention Network to Improve Long-Tail Performance for Shop Search in E-Commerce. In KDD'20. 3405--3415.

Digital Library

[26]

Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. 2017. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507 (2017).

[27]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI'09. 452--461.

Digital Library

[28]

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In ESWC'18. Springer, 593--607.

[29]

Yizhou Sun and Jiawei Han. 2012. Mining Heterogeneous Information Networks: Principles and Methodologies .Morgan and Claypool Publishers.

[30]

Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. 2011. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. PVLDB, Vol. 4, 11 (2011), 992--1003.

Digital Library

[31]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. NeurIPS, Vol. 30 (2017), 5998--6008.

[32]

Petar Velivc ković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).

[33]

Hongwei Wang, Fuzheng Zhang, Mengdi Zhang, Jure Leskovec, Miao Zhao, Wenjie Li, and Zhongyuan Wang. 2019 c. Knowledge-aware graph neural networks with label smoothness regularization for recommender systems. In KDD'19. 968--977.

Digital Library

[34]

Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. 2019 d. Knowledge graph convolutional networks for recommender systems. In WWW'19 . 3307--3313.

Digital Library

[35]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019 a. Kgat: Knowledge graph attention network for recommendation. In KDD'19 . 950--958.

Digital Library

[36]

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019 b. Heterogeneous graph attention network. In WWW'19 .

Digital Library

[37]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018a. How Powerful are Graph Neural Networks?. In ICLR'18 .

[38]

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018b. Representation Learning on Graphs with Jumping Knowledge Networks. ICML (2018), 5449--5458.

[39]

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575 (2014).

[40]

Carl Yang, Aditya Pal, Andrew Zhai, Nikil Pancha, Jiawei Han, Charles Rosenberg, and Jure Leskovec. 2020 b. MultiSage: Empowering GCN with Contextualized Multi-Embeddings on Web-Scale Multipartite Networks. In KDD'20. 2434--2443.

[41]

Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2020 c. Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark. TKDE (2020).

[42]

Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. 2020 a. Understanding negative sampling in graph representation learning. In KDD'20 . 1666--1676.

Digital Library

[43]

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. 2019. Graph Transformer Networks. In NeurIPS'19 .

[44]

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019. Heterogeneous graph neural network. In KDD'19. 793--803.

Digital Library

[45]

Shichao Zhu, Chuan Zhou, Shirui Pan, Xingquan Zhu, and Bin Wang. 2019. Relation structure-aware heterogeneous graph neural network. In ICDM'19 .

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Li JGuo XJiao PWang W(2025)Learning accurate neighborhood- and self-information for higher-order relation prediction in Heterogeneous Information NetworksNeurocomputing10.1016/j.neucom.2024.128739613(128739)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128739
Li YZang GSong CYuan X(2025)A Universal Adaptive Algorithm for Graph Anomaly DetectionInformation Processing & Management10.1016/j.ipm.2024.10390562:1(103905)Online publication date: Jan-2025
https://doi.org/10.1016/j.ipm.2024.103905
Show More Cited By

Index Terms

Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Mathematics of computing
  1. Discrete mathematics
    1. Graph theory
      1. Graph algorithms

Recommendations

Progress towards the total domination game 3 4 -conjecture

In this paper, we continue the study of the total domination game in graphs introduced in Henning et al. (2015), where the players Dominator and Staller alternately select vertices of G . Each vertex chosen must strictly increase the number of vertices ...
Progress towards the two-thirds conjecture on locating-total dominating sets
Abstract
We study upper bounds on the size of optimum locating-total dominating sets in graphs. A set S of vertices of a graph G is a locating-total dominating set if every vertex of G has a neighbor in S, and if any two vertices outside S have distinct ...
Really straight graph drawings
GD'04: Proceedings of the 12th international conference on Graph Drawing

We study straight-line drawings of graphs with few segments and few slopes. Optimal results are obtained for all trees. Tight bounds are obtained for outerplanar graphs, 2-trees, and planar 3-trees. We prove that every 3-connected plane graph on n ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

August 2021

4259 pages

ISBN:9781450383325

DOI:10.1145/3447548

General Chairs:
Feida Zhu
Singapore Management University
,
Beng Chin Ooi
National University of Singapore
,
Chunyan Miao
Nanyang Technology University
,
Program Chairs:
Haixun Wang,
Iryna Skrypnyk,
Wynne Hsu,
Sanjay Chawla

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

KDD '21

Sponsor:

KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2021

Virtual Event, Singapore

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

105
Total Citations
View Citations
2,364
Total Downloads

Downloads (Last 12 months)656
Downloads (Last 6 weeks)75

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Li JGuo XJiao PWang W(2025)Learning accurate neighborhood- and self-information for higher-order relation prediction in Heterogeneous Information NetworksNeurocomputing10.1016/j.neucom.2024.128739613(128739)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128739
Li YZang GSong CYuan X(2025)A Universal Adaptive Algorithm for Graph Anomaly DetectionInformation Processing & Management10.1016/j.ipm.2024.10390562:1(103905)Online publication date: Jan-2025
https://doi.org/10.1016/j.ipm.2024.103905
Xu ZWu J(2025)Contrastive meta-reinforcement learning for heterogeneous graph neural architecture searchExpert Systems with Applications10.1016/j.eswa.2024.125433260(125433)Online publication date: Jan-2025
https://doi.org/10.1016/j.eswa.2024.125433
Peng JGong JZhou CZang QFang XYang KYu J(2024)KGCFRec: Improving Collaborative Filtering Recommendation with Knowledge GraphElectronics10.3390/electronics1310192713:10(1927)Online publication date: 15-May-2024
https://doi.org/10.3390/electronics13101927
陈丹(2024)Algorithm Research of Attention Mechanism in Graph Neural Network ModelModeling and Simulation10.12677/MOS.2024.13102213:01(225-238)Online publication date: 2024
https://doi.org/10.12677/MOS.2024.131022
Xue RYan MHan DTeng YTang ZYe XFan DDe V(2024)GDR-HGNN: A Heterogeneous Graph Neural Networks Accelerator Frontend with Graph Decoupling and RecouplingProceedings of the 61st ACM/IEEE Design Automation Conference10.1145/3649329.3656540(1-6)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3649329.3656540
Fu QJi YRolinger THuang H(2024)TLPGNN: A Lightweight Two-level Parallelism Paradigm for Graph Neural Network Computation on Single and Multiple GPUsACM Transactions on Parallel Computing10.1145/364471211:2(1-28)Online publication date: 8-Jun-2024
https://doi.org/10.1145/3644712
Tang JYang YWei WShi LXia LYin DHuang CBaeza-Yates RBonchi F(2024)HiGPT: Heterogeneous Graph Language ModelProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671987(2842-2853)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671987
Duan HXie CLi LBaeza-Yates RBonchi F(2024)Reserving-Masking-Reconstruction Model for Self-Supervised Heterogeneous Graph RepresentationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671719(689-700)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671719
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents