research-article

Open access

Heterogeneous Graph Transformer

Authors:

Yizhou SunAuthors Info & Claims

WWW '20: Proceedings of The Web Conference 2020

Pages 2704 - 2710

https://doi.org/10.1145/3366423.3380027

Published: 20 April 2020 Publication History

All formats PDF

Abstract

Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making it infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To model heterogeneity, we design node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm—HGSampling—for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state-of-the-art GNN baselines by 9–21 on various downstream tasks. The dataset and source code of HGT are publicly available at https://github.com/acbull/pyHGT.

References

[1]

Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling. In ICLR’18.

[2]

Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable Representation Learning for Heterogeneous Networks. In KDD ’17.

[3]

Matthias Fey and Jan Eric Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. ICLR 2019 Workshop: Representation Learning on Graphs and Manifolds (2019).

[4]

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NeurIPS’17.

[5]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR’16.

[6]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR’17.

[7]

Hang Li. 2014. Learning to Rank for Information Retrieval and Natural Language Processing, Second Edition. Morgan & Claypool Publishers. https://doi.org/10.2200/S00607ED2V01Y201410HLT026

[8]

Tie-Yan Liu. 2011. Learning to Rank for Information Retrieval. Springer. https://doi.org/10.1007/978-3-642-14267-3

Digital Library

[9]

Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In ICLR’17.

[10]

Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In ICLR’19.

[11]

Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling Relational Data with Graph Convolutional Networks. In ESWC’2018.

Digital Library

[12]

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MAS) and Applications. In WWW Companion 2015.

Digital Library

[13]

Yizhou Sun and Jiawei Han. 2012. Mining Heterogeneous Information Networks: Principles and Methodologies. Morgan & Claypool Publishers.

[14]

Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S. Yu, and Tianyi Wu. 2011. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. In VLDB ’11.

Digital Library

[15]

Yizhou Sun, Brandon Norick, Jiawei Han, Xifeng Yan, Philip S. Yu, and Xiao Yu. 2012. Integrating meta-path selection with user-guided object clustering in heterogeneous information networks. In KDD’12.

[16]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS’17.

[17]

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR’18.

[18]

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. 2019. Heterogeneous Graph Attention Network. In KDD’19. 2022–2032.

[19]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Transformers: State-of-the-art Natural Language Processing. arxiv:cs.CL/1910.03771

[20]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS’19.

[21]

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. 2019. Graph Transformer Networks. In NeurIPS’19.

[22]

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. 2019. Heterogeneous Graph Neural Network. In WWW’19.

[23]

Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Bin Shao, Rui Li, and Kuansan Wang. 2019. OAG: Toward Linking Large-scale Heterogeneous Entity Graphs. In KDD’19.

Digital Library

[24]

Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. 2019. Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks. In NeurIPS’19.

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Li JGuo XJiao PWang W(2025)Learning accurate neighborhood- and self-information for higher-order relation prediction in Heterogeneous Information NetworksNeurocomputing10.1016/j.neucom.2024.128739613(128739)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128739
Gong YLv XYuan ZHu FCai ZChen YWang ZYou X(2025)CE-DCVSI: Multimodal relational extraction based on collaborative enhancement of dual-channel visual semantic informationExpert Systems with Applications10.1016/j.eswa.2024.125608262(125608)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125608
Show More Cited By

Index Terms

Heterogeneous Graph Transformer
1. Theory of computation

Index terms have been assigned to the content through auto-classification.

Recommendations

Heterogeneous graph neural networks with denoising for graph embeddings
Abstract
With the increasing popularity of graph structures, Graph embedding, Which aims to project nodes into low dimensional space while preserving the topological structure information of graphs and the information of nodes themselves, Has ...
MSLS: Meta-graph Search with Learnable Supernet for Heterogeneous Graph Neural Networks
SSDBM '23: Proceedings of the 35th International Conference on Scientific and Statistical Database Management

In recent years, heterogeneous graph neural networks (HGNNs) have achieved excellent performance. The efficient HGNNs consist of meta-graphs and aggregation operations. Since manually designing meta-graph is an expert-dependent and time-consuming ...
Scalable Heterogeneous Graph Neural Networks for Predicting High-potential Early-stage Startups
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

It is critical and important for venture investors to find high-potential startups at their early stages. Indeed, many efforts have been made to study the key factors for the success of startups through the topological analysis of the heterogeneous ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '20: Proceedings of The Web Conference 2020

April 2020

3143 pages

ISBN:9781450370233

DOI:10.1145/3366423

Editors:
Yennun Huang
Acadmica sinica, Taiwan
,
Irwin King
The Chinese University of Hong Kong, Hong Kong
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

671
Total Citations
View Citations
20,073
Total Downloads

Downloads (Last 12 months)4,281
Downloads (Last 6 weeks)671

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Li JGuo XJiao PWang W(2025)Learning accurate neighborhood- and self-information for higher-order relation prediction in Heterogeneous Information NetworksNeurocomputing10.1016/j.neucom.2024.128739613(128739)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128739
Gong YLv XYuan ZHu FCai ZChen YWang ZYou X(2025)CE-DCVSI: Multimodal relational extraction based on collaborative enhancement of dual-channel visual semantic informationExpert Systems with Applications10.1016/j.eswa.2024.125608262(125608)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125608
Zhang ZAo XTessone CLiu GZhou MMao RLiao H(2025)Multiplex graph fusion network with reinforcement structure learning for fraud detection in online e-commerce platformsExpert Systems with Applications10.1016/j.eswa.2024.125598262(125598)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125598
Xu ZWu J(2025)Contrastive meta-reinforcement learning for heterogeneous graph neural architecture searchExpert Systems with Applications10.1016/j.eswa.2024.125433260(125433)Online publication date: Jan-2025
https://doi.org/10.1016/j.eswa.2024.125433
Li BWang HTan XLi QChen JQiu X(2025)Adaptive heterogeneous graph reasoning for relational understanding in interconnected systemsThe Journal of Supercomputing10.1007/s11227-024-06623-781:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06623-7
Wang J(2024)Deep Learning in Hematology: From Molecules to PatientsClinical Hematology International10.46989/001c.1241316:4Online publication date: 8-Oct-2024
https://doi.org/10.46989/001c.124131
Zhang JLu GYu J(2024)A Smart Contract Vulnerability Detection Method Based on Heterogeneous Contract Semantic Graphs and Pre-Training TechniquesElectronics10.3390/electronics1318378613:18(3786)Online publication date: 23-Sep-2024
https://doi.org/10.3390/electronics13183786
Ma SWang W(2024)Proactive Return Prediction in Online Fashion Retail Using Heterogeneous Graph Neural NetworksElectronics10.3390/electronics1307139813:7(1398)Online publication date: 8-Apr-2024
https://doi.org/10.3390/electronics13071398
Zhang PLin PLi DWang WQi XLi JXiong J(2024)MGACL: Prediction Drug–Protein Interaction Based on Meta-Graph Association-Aware Contrastive LearningBiomolecules10.3390/biom1410126714:10(1267)Online publication date: 8-Oct-2024
https://doi.org/10.3390/biom14101267
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents