research-article

G-Prompt: : Graphon-based Prompt Tuning for graph classification

Authors:

Jianhua WuAuthors Info & Claims

Volume 61, Issue 3

https://doi.org/10.1016/j.ipm.2023.103639

Published: 01 May 2024 Publication History

Abstract

Prompt tuning has been demonstrated to be effective in exploiting pre-trained models (PTMs) to perform downstream tasks. Motivated by this trend, researchers explored prompt tuning methods to leverage graph PTMs in downstream tasks related to graph classification. However, the inherent non-Euclidean and abstract characteristics of graph data present a set of challenging issues, which encompass the generation of graph-level prompts with accurate task-related knowledge, the enhancement of prompt adaptability to downstream tasks, and the prediction based on graph-level prompts. To address these issues, we propose Graphon-based Prompt Tuning (G-Prompt) as a systemic solution. G-Prompt consists of two elaborate modules. Specifically, we present a graph-level prompt generation (GP) module that describes the knowledge of downstream tasks by estimating graphons and generates graph-level prompts based on the knowledge. Subsequently, GP module performs prompt ensembling steps and optimization based on downstream tasks to ensure the knowledge distribution of the prompts and their adaptability to downstream tasks. Furthermore, we propose a graph answer (GA) module to make accurate predictions based on multiple graph-level prompts. GA module models the conditional probability that the input graph belongs to the class corresponding to the prompt based on different prompts, respectively. It then selects the class with the highest probability as the prediction. Extensive experiments with 6 real-world datasets show that G-Prompt achieves state-of-the-art performance, outperforming compared methods by an average of 5%.

Highlights

•

Introducing graphon theory to generate graph-level prompts with knowledge.

•

Modeling conditional probabilities for prediction based on graph-level prompts.

•

Combining with existing graph pre-trained models for better performance.

References

[1]

Brown, T., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. In Proceedings of the 34th advances in neural information processing systems, vol. 33 (pp. 1877–1901).

[2]

Carrington P.J., Scott J., Wasserman S., Models and methods in social network analysis, vol. 28, Cambridge University Press, 2005.

[3]

Chatterjee S., Matrix estimation by universal singular value thresholding, The Annals of Statistics 43 (1) (2015) 177–214.

[4]

Chen, J., Ma, T., & Xiao, C. (2018). Fastgcn: fast learning with graph convolutional networks via importance sampling. In 6th international conference on learning representations.

[5]

Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., et al. (2022). Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM web conference 2022 (pp. 2778–2788).

[6]

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (Long and short papers) (pp. 4171–4186).

[7]

Diaconis P., Janson S., Graph limits and exchangeable random graphs, 2007, arXiv preprint arXiv:0712.2749.

[8]

Fadaee, M., Bisazza, A., & Monz, C. (2017). Data augmentation for low-resource neural machine translation. In Proceedings of the 55th annual meeting of the Association for Computational Linguistics (Volume 2: Short papers) (pp. 567–573).

[9]

Frieze A., Kannan R., Quick approximation to matrices and applications, Combinatorica 19 (2) (1999) 175–220.

[10]

Gao, T., Fisch, A., & Chen, D. (2021). Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (Volume 1: Long papers) (pp. 3816–3830).

[11]

Goldenberg A., Zheng A.X., Fienberg S.E., Airoldi E.M., et al., A survey of statistical network models, Foundations and Trends® in Machine Learning 2 (2) (2010) 129–233.

[12]

Han X., Zhao W., Ding N., Liu Z., Sun M., Ptr: Prompt tuning with rules for text classification, AI Open 3 (2022) 182–192.

[13]

Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., et al. (2022). Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification. In Proceedings of the 60th annual meeting of the Association for Computational Linguistics (Volume 1: Long papers) (pp. 2225–2240).

[14]

Hu, Z., Dong, Y., Wang, K., Chang, K. W., & Sun, Y. (2020). Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1857–1867).

[15]

Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., et al. (2019). Strategies for pre-training graph neural networks. In International conference on learning representations.

[16]

Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., et al. (2022). Visual prompt tuning. In Proceedings of 17th European conference on computer vision (pp. 709–727).

[17]

Jiang G., Liu S., Zhao Y., Sun Y., Zhang M., Fake news detection via knowledgeable prompt learning, Information Processing & Management 59 (5) (2022).

[18]

Jin, W., Ma, Y., Liu, X., Tang, X., Wang, S., & Tang, J. (2020). Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 66–74).

[19]

Kipf T.N., Welling M., Variational graph auto-encoders, 2016, arXiv preprint arXiv:1611.07308.

[20]

Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing (pp. 3045–3059).

[21]

Li, X. L., & Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (Volume 1: Long papers) (pp. 4582–4597).

[22]

Liu, Z., Yu, X., Fang, Y., & Zhang, X. (2023). GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks. In Proceedings of the ACM web conference 2023 (pp. 417–428).

[23]

Liu P., Yuan W., Fu J., Jiang Z., Hayashi H., Neubig G., Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys 55 (9) (2023) 1–35.

[24]

Lovász L., Large networks and graph limits, vol. 60, American Mathematical Soc., 2012.

[25]

Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., & Zhang, C. (2018). Adversarially regularized graph autoencoder for graph embedding. In Proceedings of the 27th international joint conference on artificial intelligence and the 23rd European conference on artificial intelligence (pp. 2609–2615).

[26]

Qiu X., Sun T., Xu Y., Shao Y., Dai N., Huang X., Pre-trained models for natural language processing: A survey, Science China Technological Sciences 63 (10) (2020) 1872–1897.

[27]

Radford A., Kim J.W., et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763.

[28]

Radford A., Narasimhan K., Salimans T., Sutskever I., et al., Improving language understanding by generative pre-training, OpenAI, 2018.

[29]

Sanh, V., Webson, A., et al. (2022). Multitask Prompted Training Enables Zero-Shot Task Generalization. In The 10th international conference on learning representations.

[30]

Sun, F.-Y., Hoffmann, J., Verma, V., & Tang, J. (2020). Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR.

[31]

Sun, M., Zhou, K., He, X., Wang, Y., & Wang, X. (2022). Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining (pp. 1717–1727).

[32]

Tao Z., Wei Y., Wang X., He X., Huang X., Chua T.-S., Mgat: Multimodal graph attention network for recommendation, Information Processing & Management 57 (5) (2020).

[33]

Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2018). Graph attention networks. In 6th international conference on learning representations.

[34]

Velickovic, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., & Hjelm, R. D. (2019). Deep Graph Infomax. In 7th international conference on learning representations.

[35]

Wale N., Watson I.A., Karypis G., Comparison of descriptor spaces for chemical compound retrieval and classification, Knowledge and Information Systems 14 (2008) 347–375.

[36]

Wang F., Zhu L., Li J., Chen H., Zhang H., Unsupervised soft-label feature selection, Knowledge-Based Systems 219 (2021).

[37]

Welling, M., & Kipf, T. N. (2017). Semi-supervised classification with graph convolutional networks. In The 5th international conference on learning representations.

[38]

Wu F., Souza A., Zhang T., Fifty C., Yu T., Weinberger K., Simplifying graph convolutional networks, in: International conference on machine learning, PMLR, 2019, pp. 6861–6871.

[39]

Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2019). How powerful are graph neural networks?. In International conference on learning representations.

[40]

Yanardag, P., & Vishwanathan, S. (2015). Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1365–1374).

[41]

Yang Y., Xie G., Efficient identification of node importance in social networks, Information Processing & Management 52 (5) (2016) 911–922.

[42]

Yi Z., Ounis I., Macdonald C., Contrastive graph prompt-tuning for cross-domain recommendation, ACM Transactions on Information Systems (2023).

[43]

You, Y., Chen, T., Shen, Y., & Wang, Z. (2021). Graph contrastive learning automated. In Proceedings of the 38th international conference on machine learning (pp. 12121–12132).

[44]

You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., & Shen, Y. (2020). Graph contrastive learning with augmentations. In Proceedings of the 34th advances in neural information processing systems, vol. 33 (pp. 5812–5823).

[45]

You Y., Chen T., Wang Z., Shen Y., When does self-supervision help graph convolutional networks?, in: International conference on machine learning, PMLR, 2020, pp. 10871–10880.

[46]

Yu L., Sun L., Du B., Liu C., Lv W., Xiong H., Heterogeneous graph representation learning with relation awareness, IEEE Transactions on Knowledge and Data Engineering (2022).

[47]

Yuan, W., Neubig, G., & Liu, P. (2021). Bartscore: Evaluating generated text as text generation. In Proceedings of the 35th advances in neural information processing systems, vol. 34 (pp. 27263–27277).

[48]

Zhang, M., Cui, Z., Neumann, M., & Chen, Y. (2018). An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1 (1), (pp. 4438–4445).

[49]

Zhang Z., Cui P., Zhu W., Deep learning on graphs: A survey, IEEE Transactions on Knowledge and Data Engineering 34 (1) (2020) 249–270.

[50]

Zhang, Z., Liu, Q., Wang, H., Lu, C., & Lee, C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. 34, In Proceedings of the 35th advances in neural information processing systems, vol. 34 (pp. 15870–15882).

[51]

Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022a). Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16816–16825).

[52]

Zhou K., Yang J., Loy C.C., Liu Z., Learning to prompt for vision-language models, International Journal of Computer Vision 130 (9) (2022) 2337–2348.

Cited By

Zhou DPang ZLi WSerra ESpezzano F(2024)Learning Cross-modal Knowledge Reasoning and Heuristic-prompt for Visual-language NavigationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679740(3453-3462)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679740

Index Terms

G-Prompt: Graphon-based Prompt Tuning for graph classification
1. Computing methodologies
  1. Machine learning
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Active learning
      2. Semi-supervised learning

Index terms have been assigned to the content through auto-classification.

Recommendations

All in One: Multi-Task Prompting for Graph Neural Networks
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Recently, "pre-training and fine-tuning'' has been adopted as a standard workflow for many graph tasks since it can take general graph knowledge to relieve the lack of graph annotations from each application. However, graph tasks with node level, edge ...
Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Prompt tuning has enhanced the performance of Pre-trained Language Models for multi-task learning in few-shot scenarios. However, existing studies fail to consider that the prompts among different layers in Transformer are different due to the diverse ...
Structured self-attention architecture for graph-level representation learning
Highlights
- We develop a Structured Self-attention Architecture for graph-level representation. Compared with previous GNN variants, the architecture proposed in this ...
Abstract
Recently, graph neural networks (GNNs) have shown to be effective in learning representative graph features. However, current pooling-based strategies for graph classification lack efficient utilization of graph representation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 61, Issue 3

May 2024

1388 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 May 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou DPang ZLi WSerra ESpezzano F(2024)Learning Cross-modal Knowledge Reasoning and Heuristic-prompt for Visual-language NavigationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679740(3453-3462)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679740

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents