Tag that issue: applying API-domain labels in issue tracking systems
Authors: 
Fabio Santos, Joseph Vargovich, Bianca Trinkenreich, Italo Santos, Jacob Penney, Ricardo Britto, João Felipe Pimentel, Igor Wiese, 
Igor Steinmacher, Anita Sarma, 
Marco A. GerosaAuthors Info & Claims
Abstract
Labeling issues with the skills required to complete them can help contributors to choose tasks in Open Source Software projects. However, manually labeling issues is time-consuming and error-prone, and current automated approaches are mostly limited to classifying issues as bugs/non-bugs. We investigate the feasibility and relevance of automatically labeling issues with what we call “API-domains,” which are high-level categories of APIs. Therefore, we posit that the APIs used in the source code affected by an issue can be a proxy for the type of skills (e.g., DB, security, UI) needed to work on the issue. We ran a user study (n=74) to assess API-domain labels’ relevancy to potential contributors, leveraged the issues’ descriptions and the project history to build prediction models, and validated the predictions with contributors (n=20) of the projects. Our results show that (i) newcomers to the project consider API-domain labels useful in choosing tasks, (ii) labels can be predicted with a precision of 84% and a recall of 78.6% on average, (iii) the results of the predictions reached up to 71.3% in precision and 52.5% in recall when training with a project and testing in another (transfer learning), and (iv) project contributors consider most of the predictions helpful in identifying needed skills. These findings suggest our approach can be applied in practice to automatically label issues, assisting developers in finding tasks that better match their skills.
References
[1]
Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. in Proceedings of the 1993 ACM SIGMOD international conference on management of data, pp 207–216
[2]
Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc Y-G (2008) Is it a bug or an enhancement? a text-based approach to classify change requests. in Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds, pp 304–318
[3]
API definition (2022) Available: https://languages.oup.com/google-dictionary-en/
[4]
Balali S, Steinmacher I, Annamalai U, Sarma A, Gerosa MA (2018) Newcomer’ barriers. . . is that all? an analysis of mentors’ and newcomers’ barriers in OSS projects. Comput Supported Coop Work 27(3-6):679–714
[5]
Barcomb A, Stol K, Fitzgerald B, Riehle D (2020) Managing episodic volunteers in free/libre/open source software communities. IEEE Trans Softw Eng:1–1
[6]
Behl D, Handa S, Arora A (2014) A bug mining tool to identify and analyze security bugs using naive bayes and tf-idf. in 2014 International Conference on Reliability Optimization and Information Technology (ICROIT). IEEE, pp 294–299
[7]
Bettenburg N, Just S, Schröter A, WeißC, Premraj R, Zimmermann T (2007) Quality of bug reports in eclipse. in Proceedings of the 2007 OOPSLA workshop on eclipse technology exchange, ser. eclipse ’07. New York, NY, USA: ACM, pp 21–25
[8]
Blanco A, Casillas A, Pérez A, and de Ilarraza AD Multi-label clinical document classification: impact of label-density Expert Syst Appl 2019 138
[9]
Charte F, Rivera AJ, del Jesus MJ, and Herrera F Mlsmote: approaching imbalanced multilabel learning through synthetic instance generation Knowl-Based Syst 2015 89 385-397
[10]
Devlin J, Chang M-W, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. in NAACL,
[11]
Ducasse S and Pollet D Software architecture reconstruction: a process-oriented taxonomy IEEE Trans Softw Eng 2009 35 4 573-591
[12]
El Zanaty F, Rezk C, Lijbrink S, van Bergen W, Côté M, McIntosh S (2020) Automatic recovery of missing issue type labels. IEEE Softw
[13]
Fast bert repository (2021) Available: https://github.com/utterworks/fast-bert
[14]
Feng Y, Jones J, Chen Z, Fang C (2018) An empirical study on software failure classification with multi-label and problem-transformation techniques. in 2018 IEEE 11th International Conference on Software Testing, verification and validation (ICST). IEEE, pp 320–330
[15]
Ferreira Moreno M, Sousa Dos Santos WH, Costa Mesquita Santos R, Fontoura De Gusmao Cerqueira R (2018) Supporting knowledge creation through has: the hyperknowledge annotation system. in 2018 IEEE International Symposium on Multimedia (ISM), 239–246
[16]
Flach PA, Kull M (2015) Precision-recall-gain curves: Pr analysis done right. in NIPS 15
[17]
Goel E, Abhilasha E, Goel E, Abhilasha E (2017) Random forest: a review. Int J Adv Res Comput Sc Softw Eng 7(1)
[18]
Guggulothu T and Moiz SA Code smell detection using multi-label classification approach Softw Quality J 2020 28 3 1063-1086
[19]
Herrera F, Charte F, Rivera AJ, del Jesus MJ (2016) Multilabel classification: problem analysis, metrics and techniques, 1st ed. Springer publishing company, incorporated
[20]
Huang Y, Wang J, Wang S, Liu Z, Wang D, Wang Q (2021) Characterizing and predicting good first issues. in Proceedings of the 15th ACM/IEEE international symposium on Empirical Software Engineering and Measurement (ESEM), pp 1–12
[21]
Izadi M, Ganji S, Heydarnoori (2021) Topic recommendation for software repositories using multi-label classification algorithms. Empir Softw Eng 26:93
[22]
Izadi M, Heydarnoori A, and Gousios G Topic recommendation for software repositories using multi-label classification algorithms Empirical Softw Eng 2021 26 09
[23]
Izadi M, Akbari K, and Heydarnoori A Predicting the objective and priority of issue reports in software repositories Empirical Softw Eng 2022 27 2 1-37
[24]
Kallis R, Di Sorbo A, Canfora G, Panichella S (2019) Ticket tagger: machine learning driven issue classification. in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 406–409
[25]
Klock ACT, Gasparini I, Pimenta MS (2016) 5W2H framework: a guide to design, develop and evaluate the user-centered gamification. in Proceedings of the 15th Brazilian symposium on human factors in computing systems, pp 1–10
[26]
Lin B, Zampetti F, Bavota G, Di Penta M, Lanza M (2019) Pattern-based mining of opinions in q &a websites. in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pp 548–559
[27]
Lu Y, Li G, Zhao Z, Wen L, Jin Z (2017) Learning to infer API mappings from API documents. in International conference on knowledge science, engineering and management. Springer, pp 237–248
[28]
Nam J, Pan SJ, Kim S (2013) Transfer defect learning. in 2013 35th international conference on software engineering (ICSE). IEEE, pp 382–391
[29]
Ohno T How the toyota production system was created Japanese Econ Studies 1982 10 4 83-101
[30]
Pacaiova H (2015) Analysis and identification of nonconforming products by 5W2H method. Center for Quality
[31]
Park Y, Jensen C (2009) Beyond pretty pictures: examining the benefits of code visualization for open source newcomers. in Proceedings of the 5th IEEE international workshop on visualizing software for understanding and analysis, ser. VISSOFT ’09. IEEE, pp 3–10
[32]
Pereira RB, Plastino A, Zadrozny B, and Merschmann LH Correlation analysis of performance measures for multi-label classification Inf Process Manag 2018 54 3 359-369
[33]
Perez Q, Jean P-A, Urtado C, Vauttier S (2021) Bug or not bug? that is the question. in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, pp 47–58
[34]
Petkovic D, Sosnick-Pérez M, Okada K, Todtenhoefer R, Huang S, Miglani N, Vigil A (2016) Using the random forest classifier to assess and predict student learning of software engineering teamwork. in 2016 IEEE Frontiers in Education Conference (FIE). IEEE, pp 1–7
[35]
Pingclasai N, Hata H, Matsumoto K-I (2013) Classifying bug reports to bugs and other requests using topic modeling. in 2013 20Th asia-pacific software engineering conference (APSEC), vol 2. IEEE, pp 13–18
[36]
Pinto GHL, Figueira Filho F, Steinmacher I, Gerosa MA (2017) Training software engineers using open-source software: the professors’ perspective. in 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE &T). IEEE, pp 117–121
[37]
Pinto G, Steinmacher I, Gerosa MA (2016) More common than you think: an in-depth study of casual contributors. in IEEE 23rd international conference on software analysis, evolution, and reengineering, SANER 2016, Suita, Osaka, Japan, 14-18 March 2016 - vol 1, pp 112–123
[38]
Pushphavathi T, Suma V, Ramaswamy V (2014) A novel method for software defect prediction: hybrid of fcm and random forest. in 2014 International Conference on Electronics and Communication Systems (ICECS). IEEE, pp 1–5
[39]
Qiu D, Li B, and Leung H Understanding the API usage in Java Inf Softw Technol 2016 73 81-100
[40]
Ramos J, et al (2003) Using TF-IDF to determine word relevance in document queries. in Proceedings of the first instructional conference on machine learning, vol 242. Piscataway, NJ, pp 133–142
[41]
Ravichandiran S (2021) Getting started with google BERT: build and train state-of-the-art natural language processing models using BERT. Packt Publishing Ltd
[42]
Romano J, Kromrey J, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys? in Annual meeting of the florida association of institutional research, pp 1–3
[43]
Saito T and Rehmsmeier M The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets PloS One 2015 10 3
[44]
Santos F, Trinkenreich B, Nicolati Pimentel JF, Wiese I, Steinmacher I, Sarma A, Gerosa M (2022) How to choose a task? mismatches in perspectives of newcomers and existing contributors. Empirical Softw Eng Meas
[45]
Santos I, Wiese I, Steinmacher I, Sarma A, Gerosa MA (2022) Hits and misses: newcomers’ ability to identify skills needed for OSS tasks. in 2022 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 174–183
[46]
Santos F, Wiese I, Trinkenreich B, Steinmacher I, Sarma A, Gerosa MA (2021) Can i solve it? identifying apis required to complete OSS tasks. in 2021 IEEE/ACM 18th international conference on Mining Software Repositories (MSR). IEEE, pp 346–257
[47]
Sarma A, Gerosa MA, Steinmacher I, Leano R (2016) Training the future workforce through task curation in an OSS ecosystem. in Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp 932–935
[48]
Satapathy SM, Acharya BP, and Rath SK Early stage software effort estimation using random forest technique based on use case points IET Softw 2016 10 1 10-17
[49]
Savidis A, Savaki C (2021) Software architecture mining from source code with dependency graph clustering and visualization. in IVAPP, 12
[50]
Seah C-W, Tsang IW, and Ong Y-S Transfer ordinal label learning IEEE Trans Neural Netw Learn Syst 2013 24 11 1863-1876
[51]
Sheskin D Handbook of parametric and nonparametric statistical procedures 2020 5 Chapman & Hall
[52]
Sokolova M and Lapalme G A systematic analysis of performance measures for classification tasks Inf Process Manag 2009 45 4 427-437
[53]
spacy industrial-strength natural language processing (2021) https://spacy.io/. Accessed 04 Oct 2021
[54]
Stanik C, Montgomery L, Martens D, Fucci D, Maalej W (2018) A simple nlp-based approach to support onboarding and retention in open source communities. in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 172–182
[55]
Steinmacher I, Silva MAG, Gerosa MA, and Redmiles DF A systematic literature review on the barriers faced by newcomers to open source software projects Inf Softw Technol 2015 59 67-85
[56]
Steinmacher I, Conte TU, Gerosa MA (2015) Understanding and supporting the choice of an appropriate task to start with in open source software communities. in 2015 48th Hawaii international conference on system sciences. IEEE, pp 5299–5308
[57]
Steinmacher I, Conte T, Gerosa MA, Redmiles D (2015) Social barriers faced by newcomers placing their first contribution in open source software projects. in Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, ser. CSCW’15. New York, NY, USA: Association for computing machinery, pp 1379–1392
[58]
Steinmacher I, Treude C, Gerosa MA (2018) Let me in: guidelines for the successful onboarding of newcomers to open source projects. IEEE Softw, vol 36(4):41–49
[59]
Strauss A and Corbin J Basics of qualitative research techniques 1998 Thousand oaks, CA Sage publications
[60]
Szumilas M Explaining odds ratios J Canadian Acad Child Adolescent Psych 2010 19 3 227
[61]
Tantithamthavorn C, McIntosh S, Hassan AE, and Matsumoto K The impact of automated parameter optimization on defect prediction models IEEE Trans Softw Eng 2019 45 7 683-711
[62]
Transformers documentation (2021). Available: https://huggingface.co/docs/transformers/index
[63]
Tsoumakas G, Katakis I, Vlahavas I (2009) Mining multi-label data. Data Mining Knowl Discover Handbook:667–685
[64]
Uddin G, Khomh F (2019) Automatic mining of opinions expressed about apis in stack overflow. IEEE Trans Softw Eng:1–1
[65]
Vadlamani SL, Baysal O (2020) Studying software developer expertise and contributions in stack overflow and GitHub. in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 312–323
[66]
Van Gompel M, Van Den Bosch A (2016) Efficient n-gram, skipgram and flexgram modelling with colibri core. J Open Res Softw 4(1)
[67]
Vargas-Baldrich S, Linares-Vásquez M, Poshyvanyk D (2015) Automated tagging of software projects using bytecode and dependencies. in 2015 30th IEEE/ACM international conference on Automated Software Engineering (ASE). IEEE, pp 289–294
[68]
Vaz L, Steinmacher I, Marczak S (2019) An empirical study on task documentation in software crowdsourcing on topcoder. in 2019 ACM/IEEE 14th International Conference on Global Software Engineering (ICGSE). IEEE, pp 48–57
[69]
Wang J, Sarma A (2011) Which bug should i fix: helping new developers onboard a new project. In Proceedings of the 4th international workshop on cooperative and human aspects of software engineering, ACM, pp 76–79
[70]
Wang J, Zhang X, Chen L (2021) How well do pre-trained contextual language representations recommend labels for GitHub issues?. Knowl-Based Syst 232:107476. Available: https://www.sciencedirect.com/science/article/pii/S0950705121007383
[71]
Wiese IS, Ré R, Steinmacher I, Kuroda RT, Oliva GA, Treude C, and Gerosa MA Using contextual information to predict co-changes J Syst Softw 2017 128 220-235
[72]
Xia X, Lo D, Wang X, Zhou B (2013) Tag recommendation in software information sites. in 2013 10th Working conference on mining software repositories (MSR). IEEE, pp 287–296
[73]
You Y, Li J, Reddi S, Hseu J, Kumar S, Bhojanapalli S, Song X, Demmel J, Keutzer K, Hsieh C-J (2020) Large batch optimization for deep learning: training bert in 76 minutes. in International conference on learning representations. Available: https://openreview.net/forum?id=Syx4wnEtvH
[74]
Zhang M-L and Zhou Z-H Ml-knn: a lazy learning approach to multi-label learning Pattern Recogni 2007 40 7 2038-2048
[75]
Zhou Y, Tong Y, Gu R, and Gall H Combining text mining and data mining for bug report classification J Softw Evol Process 2016 28 3 150-176
[76]
Zhu Y, Pan M, Pei Y, Zhang T (2019) A bug or a suggestion? an automatic way to label issues. arXiv:1909.00934
[77]
Zimmermann T, Premraj R, Bettenburg N, Just S, Schroter A, Weiss C (2010) What makes a good bug report IEEE Trans Softw Eng, vol 36(5), pp 618–643
Recommendations
Skill Recommendation for New Contributors in Open-Source Software
ICSE '23: Proceedings of the 45th International Conference on Software Engineering: Companion ProceedingsSelecting an appropriate task is challenging for newcomers to Open Source Software (OSS) projects. Therefore, researchers and OSS projects have proposed strategies to label tasks (a.k.a. issues). Several approaches relying on machine learning ...
Hierarchical Multi-Label Classification with Partial Labels and Unknown Hierarchy
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementHierarchical multi-label classification aims at learning a multi-label classifier from a dataset whose labels are organized into a hierarchical structure. To the best of our knowledge, we propose for the first time the problem of finding a multi-label ...
Confidence-based Weighted Loss for Multi-label Classification with Missing Labels
ICMR '20: Proceedings of the 2020 International Conference on Multimedia RetrievalThe problem of multi-label classification with missing labels (MLML) is a common challenge that is prevalent in several domains, e.g. image annotation and auto-tagging. In multi-label classification, each instance may belong to multiple class labels ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 31 August 2023
Accepted: 03 April 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in