Network Embedding With Dual Generation Tasks
Authors: 
Na Li, Jie Liu, Zhicheng He, Chunhai Zhang, Jiaying XieAuthors Info & Claims
Pages 7303 - 7315
Abstract
We study the problem of Network Embedding (NE) for content-rich networks. NE models aim to learn efficient low-dimensional dense vectors for network vertices which are crucial to many network analysis tasks. The core problem of content-rich network embedding is to learn and integrate the semantic information conveyed by network structure and node content. In this paper, we propose a general end-to-end model, <bold>D</bold>ual <bold>GE</bold>nerative <bold>N</bold>etwork <bold>E</bold>mbedding (DGENE), to leverage the complementary information of network structure and content. In this model, each vertex is regarded as an object with two modalities: node identity and textual content. Then we formulate two dual generation tasks, Node Identification (NI) which recognizes nodes’ identities given their contents, and Content Generation (CG) which generates textual contents given the nodes’ identities. We develop specific Content2Node and Node2Content models for the two tasks. Under the DGENE framework, the two dual models are learned by sharing and integrating intermediate layers. Extensive experimental results show that our model yields a significant performance gain compared to the state-of-the-art NE methods. Moreover, our model has an interesting and useful byproduct, that is, a component of our model can generate texts and nodes, which is potentially useful for many tasks.
References
[1]
C. Yang, Z. Liu, D. Zhao, M. Sun, and E. Y. Chang, “Network representation learning with rich text information,” in Proc. Int. Joint Conf. Artif. Intell., 2015, pp. 2111–2117.
[2]
C. Tu, H. Liu, Z. Liu, and M. Sun, “CANE: Context-aware network embedding for relation modeling,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2017, pp. 1722–1731.
[3]
J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “LINE: Large-scale information network embedding,” in Proc. Int. Conf. World Wide Web, 2015, pp. 1067–1077.
[4]
J. Tang, M. Qu, and Q. Mei, “PTE: Predictive text embedding through large-scale heterogeneous text networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2015, pp. 1165–1174.
[5]
X. Sun, J. Guo, X. Ding, and T. Liu, “A general framework for content-enhanced network representation learning,” 2016,.
[6]
R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2018, pp. 974–983.
[7]
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” CoRR, vol. abs/1609.02907, 2016.
[8]
Q. Long, Y. Wang, L. Du, G. Song, Y. Jin, and W. Lin, “Hierarchical community structure preserving network embedding: A subspace approach,” in Proc. ACM Int. Conf. Inf. Knowl. Manage., 2019, pp. 409–418.
[9]
M. Wu, S. Pan, X. Zhu, C. Zhou, and L. Pan, “Domain-adversarial graph neural networks for text classification,” in Proc. IEEE Int. Conf. Data Mining, 2019, pp. 648–657.
[10]
Y. Wang et al., “Tag2Gauss: Learning tag representations via gaussian distribution in tagged networks,” in Proc. Int. Joint Conf. Artif. Intell., 2019, pp. 3799–3805.
[11]
B. Perozzi, R. Al-Rfou, and S. Skiena, “DeepWalk: Online learning of social representations,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2014, pp. 701–710.
[12]
A. Grover and J. Leskovec, “Node2Vec: Scalable feature learning for networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2016, pp. 855–864.
[13]
S. Cao, W. Lu, and Q. Xu, “GraRep: Learning graph representations with global structural information,” in Proc. ACM Int. Conf. Inf. Knowl. Manage., 2015, pp. 891–900.
[14]
X. Wang, P. Cui, J. Wang, J. Pei, W. Zhu, and S. Yang, “Community preserving network embedding,” in Proc. AAAI Conf. Artif. Intell., 2017, pp. 203–209.
[15]
C. Tu, W. Zhang, Z. Liu, and M. Sun, “Max-margin DeepWalk: Discriminative learning of network representation,” in Proc. Int. Joint Conf. Artif. Intell., 2016, pp. 3889–3895.
[16]
J. Chen, Q. Zhang, and X. Huang, “Incorporate group information to enhance network embedding,” in Proc. ACM Int. Conf. Inf. Knowl. Manage., 2016, pp. 1901–1904.
[17]
L. Du, Z. Lu, Y. Wang, G. Song, Y. Wang, and W. Chen, “Galaxy network embedding: A hierarchical community structure preserving approach,” in Proc. Int. Joint Conf. Artif. Intell., 2018, pp. 2079–2085.
[18]
S. Wang, C. C. Aggarwal, J. Tang, and H. Liu, “Attributed signed network embedding,” in Proc. ACM Int. Conf. Inf. Knowl. Manage., 2017, pp. 137–146.
[19]
H. Gao and H. Huang, “Deep attributed network embedding,” in Proc. Int. Joint Conf. Artif. Intell., 2018, pp. 3364–3370.
[20]
Y. Cen, X. Zou, J. Zhang, H. Yang, J. Zhou, and J. Tang, “Representation learning for attributed multiplex heterogeneous network,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2019, pp. 1358–1368.
[21]
S. Bandyopadhyay, N. Lokesh, and M. N. Murty, “Outlier aware network embedding for attributed networks,” in Proc. AAAI Conf. Artif. Intell., 2019, pp. 12–19.
[22]
Y. Shi, Q. Zhu, F. Guo, C. Zhang, and J. Han, “Easing embedding learning by comprehensive transcription of heterogeneous information networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2018, pp. 2190–2199.
[23]
B. Hu, Y. Fang, and C. Shi, “Adversarial learning on heterogeneous information networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2019, pp. 120–129.
[24]
Y. Lu, C. Shi, L. Hu, and Z. Liu, “Relation structure-aware heterogeneous information network embedding,” in Proc. AAAI Conf. Artif. Intell., 2019, pp. 4456–4463.
[25]
W. Yu, W. Cheng, C. C. Aggarwal, K. Zhang, H. Chen, and W. Wang, “NetWalk: A flexible deep embedding approach for anomaly detection in dynamic networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2018, pp. 2672–2681.
[26]
T. Li, J. Zhang, P. S. Yu, Y. Zhang, and Y. Yan, “Deep dynamic network embedding for link prediction,” IEEE Access, vol. 6, pp. 29 219–29 230, 2018.
[27]
Z. He, J. Liu, N. Li, and Y. Huang, “Learning network-to-network model for content-rich network embedding,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2019, pp. 1037–1045.
[28]
W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2017, pp. 1024–1034.
[29]
P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” CoRR, vol. abs/1710.10903, 2017.
[30]
J. Liu, Z. He, L. Wei, and Y. Huang, “Content to node: Self-translation network embedding,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2018, pp. 1794–1802.
[31]
J. L. Elman, “Finding structure in time,” Cogn. Sci., vol. 14, no. 2, pp. 179–211, 1990.
[32]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[33]
A. Graves, “Generating sequences with recurrent neural networks,” 2013,.
[34]
S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2017, pp. 4835–4839.
[35]
X. Ma and E. H. Hovy, “End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2016, pp. 1064–1074.
[36]
Q. Xia, Z. Li, and M. Zhang, “A syntax-aware multi-task learning framework for chinese semantic role labeling,” in Proc. Conf. Empir. Methods Natural Lang. Process. 9th Int. Joint Conf. Natural Lang. Process., 2019, pp. 5382–5392.
[37]
Z. Yang, D. Yang, C. Dyer, X. He, A. J. Smola, and E. H. Hovy, “Hierarchical attention networks for document classification,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2016, pp. 1480–1489.
[38]
Z. Wu, X. Dai, C. Yin, S. Huang, and J. Chen, “Improving review representations with user attention and product attention for sentiment classification,” in Proc. AAAI Conf. Artif. Intell., 2018, pp. 5989–5996.
[39]
A. Vaswani et al., “Attention is all you need,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2017, pp. 5998–6008.
[40]
K. Cho et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2014, pp. 1724–1734.
[41]
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2014, pp. 3104–3112.
[42]
O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. E. Hinton, “Grammar as a foreign language,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2015, pp. 2773–2781.
[43]
L. Dong and M. Lapata, “Language to logical form with neural attention,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2016, pp. 33–43.
[44]
D. Ma, S. Li, F. Wu, X. Xie, and H. Wang, “Exploring sequence-to-sequence learning in aspect term extraction,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2019, pp. 3538–3547.
[45]
K. Li, C. Chen, X. Quan, Q. Ling, and Y. Song, “Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2020, pp. 7056–7066.
[46]
J. Tan, X. Wan, and J. Xiao, “Abstractive document summarization with a graph-based attentional neural model,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2017, pp. 1171–1181.
[47]
P. Kouris, G. Alexandridis, and A. Stafylopatis, “Acl,” 2019, pp. 5082–5092.
[48]
J. Li, M. Luong, and D. Jurafsky, “A hierarchical neural autoencoder for paragraphs and documents,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2015, pp. 1106–1115.
[49]
W. Lei, X. Jin, M. Kan, Z. Ren, X. He, and D. Yin, “Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2018, pp. 1437–1447.
[50]
M. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, “Multi-task sequence to sequence learning,” 2015,.
[51]
H. Song and S. Park, “Korean morphological analysis with tied sequence-to-sequence multi-task model,” in Proc. Conf. Empir. Methods Natural Lang. Process. 9th Int. Joint Conf. Natural Lang. Process., 2019, pp. 1436–1441.
[52]
D. He et al., “Dual learning for machine translation,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2016, pp. 820–828.
[53]
Y. Xia, T. Qin, W. Chen, J. Bian, N. Yu, and T. Liu, “Dual supervised learning,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 3789–3798.
[54]
Y. Xia, X. Tan, F. Tian, T. Qin, N. Yu, and T. Liu, “Model-level dual learning,” in Proc. Int. Conf. Mach. Learn., 2018, pp. 5379–5388.
[55]
D. Tang, N. Duan, T. Qin, and M. Zhou, “Question answering and question generation as dual tasks,” 2017,.
[56]
S. Su, C. Huang, and Y. Chen, “Dual supervised learning for natural language understanding and generation,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2019, pp. 5472–5477.
[57]
R. Cao, S. Zhu, C. Liu, J. Li, and K. Yu, “Semantic parsing with dual learning,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2019, pp. 51–64.
[58]
M. J. Hosseini, S. B. Cohen, M. Johnson, and M. Steedman, “Duality of link prediction and entailment graph induction,” in Proc. Annu. Meeting Assoc. Comput. Linguistics, 2019, pp. 4736–4746.
[59]
Y. Kim, “Convolutional neural networks for sentence classification,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2014, pp. 1746–1751.
[60]
R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Comput., vol. 1, no. 2, pp. 270–280, 1989.
[61]
A. McCallum, K. Nigam, J. Rennie, and K. Seymore, “Automating the construction of internet portals with machine learning,” Inf. Retrieval, vol. 3, no. 2, pp. 127–163, 2000.
[62]
K. W. Lim and W. L. Buntine, “Bibliographic analysis with the citation network topic model,” in Proc. Asian Conf. Mach. Learn., 2014, pp. 142–158.
[63]
J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su, “ArnetMiner: Extraction and mining of academic social networks,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2008, pp. 990–998.
[64]
G. Corso, L. Cavalleri, D. Beaini, P. Liò, and P. Velickovic, “Principal neighbourhood aggregation for graph nets,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2020, Art. no.
[65]
M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li, “Simple and deep graph convolutional networks,” in Proc. Int. Conf. Mach. Learn., 2020, pp. 1725–1735.
[66]
Z. Wang, X. Ye, C. Wang, J. Cui, and P. S. Yu, “Network embedding with completely-imbalanced labels,” IEEE Trans. Knowl. Data Eng., vol. 33, no. 11, pp. 3634–3647, Nov. 2021.
Recommendations
Learning Network-to-Network Model for Content-rich Network Embedding
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data MiningRecently, network embedding (NE) has achieved great successes in learning low dimensional representations for network nodes and has been increasingly applied to various network analytic tasks. In this paper, we consider the representation learning ...
Greedy Routing by Network Distance Embedding
Greedy routing has been applied to both wireline and wireless networks due to its scalability of routing state and resiliency to network dynamics. In this work, we solve a fundamental problem in applying greedy routing to networks with arbitrary ...
A Dual Fusion Model for Attributed Network Embedding
Knowledge Science, Engineering and ManagementAbstractAttributed network embedding (ANE) maps nodes in network into the low-dimensional space while preserving proximities of both node attributes and network topology. Existing methods for ANE integrated node attributes and network topology by three ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1041-4347 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 July 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025