Abstract
In our work, we present the contribution of the BLUE team in the eRisk Lab task focused on identifying symptoms of depression in Reddit social media posts. The task consists of retrieving and ranking Reddit social media sentences that convey symptoms of depression from the BDI-II questionnaire. To augment our data and improve downstream models, we utilized synthetic data generated by GPT-3.5 and LLama-3 for each of the BDI-II symptoms. Our approach aimed to enrich the data with semantic diversity and emotional and anecdotal experiences that are specific to the more intimate way of sharing experiences on Reddit. We used semantic search and cosine similarity to rank the relevance of the sentences to the BDI-II symptoms. Our study compared the performance of two transformer-based models (MentalRoBERTa and a variant of MPNet) in embedding social media posts and the original/generated BDI-II responses for information retrieval. We found that using sentence embeddings from a model designed for semantic search outperformed the approach using embeddings from a model pre-trained on mental health data. Furthermore, the generated synthetic data were proved too specific for this task, the approach simply relying on the BDI-II responses had the best performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Aich, A., et al.: Towards intelligent clinically-informed language analyses of people with bipolar disorder and schizophrenia. In: Findings of EMNLP, pp. 2871–2887 (2022)
Amin, M.M., Cambria, E., Schuller, B.W.: Will affective computing emerge from foundation models and general AI? a first evaluation on ChatGPT. IEEE Intell. Syst. 38, 2 (2023)
Beck, A.T., Steer, R.A., Brown, G.: Beck depression inventory–II. Psychological assessment (1996)
Brown, T., et al.: Language models are few-shot learners. In: Proceedings of NeurIPS, vol. 33, pp. 1877–1901 (2020)
Bucur, A.M., Cosma, A., Dinu, L.P.: Early risk detection of pathological gambling, self-harm and depression using BERT. In: CLEF (Working Notes) (2021)
Bucur, A.M., Cosma, A., Rosso, P., Dinu, L.P.: It’s just a matter of time: detecting depression with time-enriched multimodal transformers. In: Kamps, J., et al. (eds.) ECIR 2023. LNCS, vol. 13980, pp. 200–215. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28244-7_13
Dai, H., et al.: ChatAug: Leveraging ChatGPT for text data augmentation. arXiv preprint arXiv:2302.13007 (2023)
De Choudhury, M., De, S.: Mental health discourse on reddit: self-disclosure, social support, and anonymity. In: Proceedings of ICWSM, pp. 71–80 (2014)
Eaton, W.W., Muntaner, C., Smith, C., Tien, A., Ybarra, M.: Center for epidemiologic studies depression scale: review and revision. Psychol. Test. Treat. Plan. Outcomes Assess. (2004)
Hamilton, M.: A rating scale for depression. J. Neurol. Neurosurg. Psychiatry 23(1), 56 (1960)
Handy, A., Mangal, R., Stead, T.S., Coffee Jr, R.L., Ganti, L.: Prevalence and impact of diagnosed and undiagnosed depression in the united states. Cureus 14(8) (2022)
Ji, S., Zhang, T., Ansari, L., Fu, J., Tiwari, P., Cambria, E.: MentalBERT: publicly available pretrained language models for mental healthcare. In: Proceedings of LREC, pp. 7184–7190 (2022)
Kroenke, K., Spitzer, R.L., Williams, J.B.: The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 16(9), 606–613 (2001)
Lee, Y.J., Lim, C.G., Choi, Y., Lm, J.H., Choi, H.J.: PERSONACHATGEN: generating personalized dialogues using GPT-3. In: Proceedings of CCGPK Workshop, pp. 29–48 (2022)
Liu, T., et al.: Detecting symptoms of depression on reddit. In: Proceedings of WebSci, pp. 174–183 (2023)
Martínez-Castaño, R., Htait, A., Azzopardi, L., Moshfeghi, Y.: Early risk detection of self-harm and depression severity using BERT-based transformers: iLab at CLEF eRisk 2020. In: CLEF (Working Notes) (2020)
Meyer, S., Elsweiler, D., Ludwig, B., Fernandez-Pichel, M., Losada, D.E.: Do we still need human assessors? Prompt-based GPT-3 user simulation in conversational AI. In: Proceedings of CUI, pp. 1–6 (2022)
Mori, S., Ignat, O., Lee, A., Mihalcea, R.: Towards algorithmic fidelity: mental health representation across demographics in synthetic vs. human-generated data. arXiv preprint arXiv:2403.16909 (2024)
Nguyen, T., Yates, A., Zirikly, A., Desmet, B., Cohan, A.: Improving the generalizability of depression detection by leveraging clinical questionnaires. In: Proceedings of ACL, pp. 8446–8459 (2022)
OpenAI: GPT-4 Technical report. arXiv (2023)
Owen, D., et al.: Enabling early health care intervention by detecting depression in users of web-based forums using language models: longitudinal analysis and evaluation. JMIR AI 2(1) (2023)
Owen, D., Camacho-Collados, J., Anke, L.E.: Towards preemptive detection of depression and anxiety in Twitter. In: Proceedings of SMM4H Workshop, pp. 82–89 (2020)
Parapar, J., Martín-Rodilla, P., Losada, D.E., Crestani, F.: Overview of eRisk 2023: Early risk prediction on the internet. In: Arampatzis, A., et al. (eds.) CLEF 2023. LNCS, vol. 14163, pp. 294–315. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-42448-9_22
Rao, G., Zhang, Y., Zhang, L., Cong, Q., Feng, Z.: MGL-CNN: a hierarchical posts representations model for identifying depressed individuals in online forums. IEEE Access 8, 32395–32403 (2020)
Skaik, R., Inkpen, D.: Using Twitter social media for depression detection in the Canadian population. In: Proceedings of AICCC, pp. 109–114 (2020)
Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: MPNet: masked and permuted pre-training for language understanding. In: Proceedings of NeurIPS, vol. 33, pp. 16857–16867 (2020)
Taori, R., et al.: Stanford alpaca: an instruction-following llama model (2023). https://github.com/tatsu-lab/stanford$_$alpaca
Trotzek, M., Koitka, S., Friedrich, C.M.: Utilizing neural networks and linguistic metadata for early detection of depression indications in text sequences. IEEE Trans. Knowl. Data Eng. 32(3), 588–601 (2018)
Uban, A.S., Chulvi, B., Rosso, P.: Multi-aspect transfer learning for detecting low resource mental disorders on social media. In: Proceedings of LREC, pp. 3202–3219 (2022)
Uban, A.S., Rosso, P.: Deep learning architectures and strategies for early detection of self-harm and depression level prediction. In: CLEF (Working Notes), vol. 2696, pp. 1–12 (2020)
Ubani, S., Polat, S.O., Nielsen, R.: ZeroShotDataAug: generating and augmenting training data with ChatGPT. arXiv preprint arXiv:2304.14334 (2023)
Wang, Y., et al.: Self-instruct: aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 (2022)
Yang, K., Ji, S., Zhang, T., Xie, Q., Ananiadou, S.: On the evaluations of ChatGPT and emotion-enhanced prompting for mental health analysis. arXiv preprint arXiv:2304.03347 (2023)
Yates, A., Cohan, A., Goharian, N.: Depression and self-harm risk assessment in online forums. In: Proceedings of EMNLP, pp. 2968–2978 (2017)
Zhang, Z., Chen, S., Wu, M., Zhu, K.: Psychiatric scale guided risky post screening for early detection of depression. In: Proceedings of IJCAI (2022)
Zhang, Z., Chen, S., Wu, M., Zhu, K.: Symptom identification for interpretable detection of multiple mental disorders on social media. In: Proceedings of EMNLP, pp. 9970–9985 (2022)
Acknowledgements
This work was supported by the POCIDIF project in Action 1.2. “Romanian Hub for Artificial Intelligence”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bucur, AM. (2024). Leveraging LLM-Generated Data for Detecting Depression Symptoms on Social Media. In: Goeuriot, L., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2024. Lecture Notes in Computer Science, vol 14958. Springer, Cham. https://doi.org/10.1007/978-3-031-71736-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-71736-9_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71735-2
Online ISBN: 978-3-031-71736-9
eBook Packages: Computer ScienceComputer Science (R0)