Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Attention-enabled adaptive Markov graph convolution

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

GNNs (Graph Neural Networks) have attracted increasing attention for their strong power on dealing with the graph structures. However, it remains a challenge to design an ideal GNN suitable for various downstream tasks. By revisiting the framework of MPNN (Message Passing Neural Network), we argue that an ideal GNN should satisfy following two conditions. First, the node embedding can absorb the knowledge from a wide range of neighbors while maintaining locality. Second, the first-order information aggregation can adapt to unknown graphs. In this paper, we first extend \(\rm{S}^{2} \rm{GC}\) to GMGC (Generalized Markov Graph Convolution), which can maintain the node locality regardless the type of the embedded diffusion kernel. Next, we embed the improved self-gating mechanism into the GMGC framework and propose a novel model named AMGC (Attention-enabled Adaptive Markov Graph Convolution), which well satisfies the above conditions. Moreover, the advantages of AMGC can be explained in the frequency domain. First, the frequency of the first-order diffusion kernel is adaptive and no longer limited to low-pass as \(\rm{S}^{2} \rm{GC}\). Second, the multi-order diffusion kernel can retain more components around the core frequency compared with FAGCN. To verify the ability of AMGC, extensive experiments are conducted, including graph regression, graph classification and semi-supervised node classification. The results show that AMGC can achieve comparable performance in all graph tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Wang F et al. (2017) Residual attention network for image classification. 2017 IEEE conference on computer vision and pattern recognition (CVPR) 6450–6458

  2. Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29:2352–2449

    Article  MathSciNet  PubMed  Google Scholar 

  3. Graves A, rahman Mohamed A, Hinton GE (2013) Speech recognition with deep recurrent neural networks. 2013 IEEE international conference on acoustics, speech and signal processing 6645–6649

  4. Chetupalli SR, Habets EAP (2023) Speaker counting and separation from single-channel noisy mixtures. IEEE ACM Trans Audio Speech Lang Process 31:1681–1692. https://doi.org/10.1109/TASLP.2023.3268572

    Article  Google Scholar 

  5. Manning CD et al. (2014) The stanford corenlp natural language processing toolkit

  6. Bai Z, Li Y, Wozniak M, Zhou M, Li D (2021) Decomvqanet: decomposing visual question answering deep network via tensor decomposition and regression. Pattern Recognit 110:107538. https://doi.org/10.1016/j.patcog.2020.107538

    Article  Google Scholar 

  7. Wozniak M, Wieczorek M, Silka J (2023) Bilstm deep neural network model for imbalanced medical data of iot systems. Future Gener Comput Syst 141:489–499. https://doi.org/10.1016/j.future.2022.12.004

    Article  Google Scholar 

  8. Yu D, Pan T (2021) Tracing knowledge diffusion of topsis: a historical perspective from citation network. Expert Syst Appl 168:114238

    Article  Google Scholar 

  9. Liao L, He X, Zhang H, Chua T-S (2018) Attributed social network embedding. IEEE Trans Knowl Data Eng 30:2257–2270

    Article  Google Scholar 

  10. Li H, Lü Q, Huang T (2019) Convergence analysis of a distributed optimization algorithm with a general unbalanced directed communication network. IEEE Transact Netw Sci Eng 6:237–248

    Article  ADS  MathSciNet  Google Scholar 

  11. Bhagat S, Cormode G, Muthukrishnan S (2011) Node classification in social networks

  12. Zhang M, Cui Z, Neumann M, Chen Y (2018) An end-to-end deep learning architecture for graph classification

  13. Dong W et al (2022) Improving performance and efficiency of graph neural networks by injective aggregation. Knowl Based Syst 254:109616. https://doi.org/10.1016/j.knosys.2022.109616

    Article  Google Scholar 

  14. Xu K, Hu W, Leskovec J, Jegelka S (2019) How powerful are graph neural networks? (OpenReview.net)

  15. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE, Precup D, Teh YW (eds) (2017). Neural message passing for quantum chemistry. (eds Precup, D. & Teh, Y. W.) Proceedings of the 34th international conference on machine learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Vol. 70 of proceedings of machine learning research, 1263–1272 (PMLR)

  16. Wu F et al. (2019) Chaudhuri, K. & Salakhutdinov, R. (eds) Simplifying graph convolutional networks. (eds Chaudhuri, K. & Salakhutdinov, R.) Proceedings of the 36th International conference on machine learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Vol. 97 of proceedings of machine learning research, pp. 6861–6871 (PMLR)

  17. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks (OpenReview.net). https://openreview.net/forum?id=SJU4ayYgl

  18. Zhu H, Koniusz P ((2021)) Simple spectral graph convolution

  19. Cui G, Zhou J, Yang C, Liu Z (2020) Adaptive graph encoder for attributed graph embedding. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

  20. Liu W et al (2020) Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans Circuits Syst Video Technol 30:23–35

    Article  MathSciNet  Google Scholar 

  21. Karam C, Hirakawa K (2018) Monte-carlo acceleration of bilateral filter and non-local means. IEEE Trans Image Process 27:1462–1474

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  22. Zhu J et al. (2020) Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds) Beyond homophily in graph neural networks: Current limitations and effective designs. (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H.) Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

  23. Velickovic P et al. (2018) Graph attention networks (OpenReview.net)

  24. Bo D, Wang X, Shi C, Shen H (2021) Beyond low-frequency information in graph convolutional networks, 3950–3957 (AAAI Press)

  25. Bruna J, Zaremba W, Szlam A, LeCun Y, Bengio Y, LeCun Y (eds) (2014)Spectral networks and locally connected networks on graphs. (eds Bengio, Y. & LeCun, Y.) 2nd International conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

  26. Hammond DK, Vandergheynst P, Gribonval R (2009) Wavelets on graphs via spectral graph theory. CoRR abs/0912.3848. arXiv:0912.3848

  27. Defferrard M, Bresson X, Vandergheynst P, Lee DD, Sugiyama M, von Luxburg U, Guyon I, Garnett R (eds) (2016) Convolutional neural networks on graphs with fast localized spectral filtering. (eds Lee, DD, Sugiyama M, von Luxburg U, Guyon I & Garnett R) Advances in neural information processing systems 29: annual conference on neural information processing systems 2016, December 5-10, 2016, Barcelona, Spain, 3837–3845

  28. Xu B, Shen H, Cao Q, Cen K, Cheng X, Kraus S (ed) (2019) Graph convolutional networks using heat kernel for semi-supervised learning. (ed. Kraus S) Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, 1928–1934 (ijcai.org)

  29. Tremblay N, Borgnat P (2014) Graph wavelets for multiscale community mining. IEEE Trans Signal Process 62(20):5227–5239. https://doi.org/10.1109/TSP.2014.2345355

    Article  ADS  MathSciNet  Google Scholar 

  30. Donnat C, Zitnik M, Hallac D, Leskovec J, Guo Y, Farooq F (eds) (2018) Learning structural node embeddings via diffusion wavelets. (eds Guo, Y. & Farooq, F.) Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, 1320–1329 (ACM)

  31. Li Q, Han Z, Wu X, McIlraith SA, Weinberger KQ (eds) (2018) Deeper insights into graph convolutional networks for semi-supervised learning. (eds McIlraith SA & Weinberger KQ) Proceedings of the Thirty-Second AAAI conference on artificial intelligence, (AAAI-18), the 30th innovative applications of artificial intelligence (IAAI-18), and the 8th AAAI symposium on educational advances in artificial intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 3538–3545 (AAAI Press)

  32. Chen Z, Chen F, Lai R, Zhang X, Lu C (2018) Rational neural networks for approximating graph convolution operator on jump discontinuities, 59–68 (IEEE Computer Society)

  33. Bianchi FM, Grattarola D, Livi LF, Alippi C (2022) Graph neural networks with convolutional arma filters. IEEE Trans Pattern Anal Mach Intell 44:3496–3507

    PubMed  Google Scholar 

  34. Hamilton WL, Ying Z, Leskovec J, Guyon I et al. (eds) (2017) Inductive representation learning on large graphs. (eds Guyon I et al.) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, 1024–1034

  35. Klicpera J, Bojchevski A, Günnemann S (2019) Predict then propagate: graph neural networks meet personalized pagerank

  36. Klicpera J, Weißenberger S, Günnemann S, Wallach HM et al. (eds) (2019) Diffusion improves graph learning. (eds Wallach HM et al.) Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 13333–13345

  37. Pei H, Wei B, Chang KC, Lei Y, Yang B (2020) Geom-gcn: Geometric graph convolutional networks (OpenReview.net)

  38. Velickovic P et al. (2020) Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds) Pointer graph networks. (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H.) Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

  39. Lee JB, Rossi RA, Kong X, Guo Y, Farooq F (eds) (2018) Graph classification using structural attention. (eds Guo, Y. & Farooq, F.) Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2018, London, UK, August 19-23, 2018, 1666–1674 (ACM)

  40. Gao H, Ji S, Teredesai A et al. (eds) (2019) Graph representation learning via hard and channel-wise attention networks. (eds Teredesai A et al.) Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, 741–749 (ACM)

  41. Wang G, Ying R, Huang J, Leskovec J (2019) Improving graph attention networks with large margin-based constraints. CoRR abs/1910.11945. arXiv:1910.11945

  42. Brody S, Alon U, Yahav E (2021) How attentive are graph attention networks? CoRR abs/2105.14491. arXiv:2105.14491

  43. He M, Wei Z, Huang Z, Xu H, Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) (2021) Bernnet: learning arbitrary graph spectral filters via bernstein approximation. (eds Ranzato M, Beygelzimer A, Dauphin YN, Liang P & Vaughan JW) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 14239–14251

  44. Vaswani A et al. (2017) Guyon I et al. (eds) Attention is all you need. (eds Guyon I et al.) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, 5998–6008

  45. Balcilar M et al. (2021) Analyzing the expressive power of graph neural networks in a spectral perspective

  46. Meyer F (1994) Topographic distance and watershed lines. Signal Process 38(1):113–125. https://doi.org/10.1016/0165-1684(94)90060-4

    Article  Google Scholar 

  47. Sen P et al (2008) Collective classification in network data. AI Mag 29(3):93–106. https://doi.org/10.1609/aimag.v29i3.2157

    Article  MathSciNet  Google Scholar 

  48. Namata G, London B, Getoor L, Huang B (2012) Query-driven active surveying for collective classification

  49. Tang J, Sun J, Wang C, Yang ZIV, JFE, Fogelman-Soulié F, Flach PA, Zaki MJ (eds) (2009) Social influence analysis in large-scale networks. (eds IV, JFE, Fogelman-Soulié, F, Flach, PA & Zaki MJ) Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining, Paris, France, June 28 - July 1, 2009, 807–816 (ACM)

Download references

Author information

Authors and Affiliations

  1. Command and Control Engineering College, Army Engineering University of PLA, Qinhuai District, Nanjing, 210001, Jiangsu, China

    Tianfeng Wang, Zhisong Pan, Guyu Hu & Yahao Hu

Authors
  1. Tianfeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhisong Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guyu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yahao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhisong Pan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Pan, Z., Hu, G. et al. Attention-enabled adaptive Markov graph convolution. Neural Comput & Applic 36, 4979–4993 (2024). https://doi.org/10.1007/s00521-023-09338-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09338-7

Keywords

Search

Navigation