research-article

Mitigating turnover with code review recommendation: balancing expertise, workload, and knowledge distribution

Authors:

Ehsan Mirsaeedi,

Peter C. RigbyAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1183 - 1195

https://doi.org/10.1145/3377811.3380335

Published: 01 October 2020 Publication History

Abstract

Developer turnover is inevitable on software projects and leads to knowledge loss, a reduction in productivity, and an increase in defects. Mitigation strategies to deal with turnover tend to disrupt and increase workloads for developers. In this work, we suggest that through code review recommendation we can distribute knowledge and mitigate turnover with minimal impacton the development process. We evaluate review recommenders in the context of ensuring expertise during review, Expertise, reducing the review workload of the core team, CoreWorkload, and reducing the Files at Risk to turnover, FaR. We find that prior work that assigns reviewers based on file ownership concentrates knowledge on a small group of core developers increasing risk of knowledge loss from turnover by up to 65%. We propose learning and retention aware review recommenders that when combined are effective at reducing the risk of turnover by -29% but they unacceptably reduce the overall expertise during reviews by -26%. We develop the Sofia recommender that suggests experts when none of the files under review are hoarded by developers, but distributes knowledge when files are at risk. In this way, we are able to simultaneously increase expertise during review with a ΔExpertise of 6%, with a negligible impact on workload of ΔCoreWorkload of 0.09%, and reduce the files at risk by ΔFaR -28%. Sofia is integrated into GitHub pull requests allowing developers to select an appropriate expert or "learner" based on the context of the review. We release the Sofia bot as well as the code and data for replication purposes.

References

[1]

Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In Proceedings of the 2013 international conference on software engineering. IEEE Press, 712--721.

Digital Library

[2]

Vipin Balachandran. 2013. Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 931--940.

Digital Library

[3]

Lingfeng Bao, Zhenchang Xing, Xin Xia, David Lo, and Shanping Li. 2017. Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 170--181.

Digital Library

[4]

Christian Bird, Alex Gourley, Prem Devanbu, Michael Gertz, and Anand Swaminathan. 2006. Mining email social networks. In Proceedings of the 2006 international workshop on Mining software repositories. ACM, 137--143.

Digital Library

[5]

Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011. Don't touch my code!: examining the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 4--14.

Digital Library

[6]

Amiangshu Bosu, Jeffrey C Carver, Christian Bird, Jonathan Orbeck, and Christopher Chockley. 2016. Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Transactions on Software Engineering 43, 1 (2016), 56--75.

Digital Library

[7]

Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of useful code reviews: An empirical study at microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146--156.

[8]

Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebastiano Panichella. 2012. Who is going to mentor newcomers in open source projects?. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 44.

Digital Library

[9]

Eleni Constantinou and Tom Mens. 2017. An empirical comparison of developer retention in the RubyGems and npm software ecosystems. Innovations in Systems and Software Engineering 13, 2--3 (2017), 101--115.

Digital Library

[10]

Michael Fagan. 2002. Design and code inspections to reduce errors in program development. In Software pioneers. Springer, 575--607.

[11]

M. E. Fagan. 1976. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15, 3 (1976), 182--211.

Digital Library

[12]

Matthieu Foucault, Marc Palyart, Xavier Blanc, Gail C Murphy, and Jean-Rémy Falleri. 2015. Impact of developer turnover on quality in open-source software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 829--841.

Digital Library

[13]

Thomas Fritz, Gail C Murphy, and Emily Hill. 2007. Does a programmer's activity indicate knowledge of code?. In Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 341--350.

Digital Library

[14]

Tudor Girba, Adrian Kuhn, Mauricio Seeberger, and Stéphane Ducasse. 2005. How developers drive software evolution. In Eighth International Workshop on Principles of Software Evolution (IWPSE'05). IEEE, 113--122.

Digital Library

[15]

Georgios Gousios. 2013. The GHTorent dataset and tool suite. In Proceedings of the 10th working conference on mining software repositories. IEEE Press, 233--236.

Digital Library

[16]

Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering. ACM, 345--355.

Digital Library

[17]

Michaela Greiler, Christian Bird, Margaret-Anne Storey, Laura MacLeod, and Jacek Czerwonka. 2016. Code Reviewing in the Trenches: Understanding Challenges, Best Practices and Tool Needs. (2016).

[18]

Christoph Hannebauer, Michael Patalas, Sebastian Stünkel, and Volker Gruhn. 2016. Automatically recommending code reviewers based on their expertise: An empirical comparison. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 99--110.

Digital Library

[19]

Mark A Huselid. 1995. The impact of human resource management practices on turnover, productivity, and corporate financial performance. Academy of management journal 38, 3 (1995), 635--672.

[20]

Daniel Izquierdo-Cortazar, Gregorio Robles, Felipe Ortega, and Jesus M Gonzalez-Barahona. 2009. Using software archaeology to measure knowledge loss in software projects due to developer turnover. In 2009 42nd Hawaii International Conference on System Sciences. IEEE, 1--10.

[21]

Gaeul Jeong, Sunghun Kim, Thomas Zimmermann, and Kwangkeun Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. Research on software analysis for error-free computing center Tech-Memo (ROSAEC MEMO 2009-006) (2009), 1--18.

[22]

Huzefa Kagdi, Maen Hammad, and Jonathan I Maletic. 2008. Who can help me with this source code change?. In 2008 IEEE International Conference on Software Maintenance. IEEE, 157--166.

[23]

Vladimir Kovalenko, Nava Tintarev, Evgeny Pasynkov, Christian Bird, and Alberto Bacchelli. 2018. Does reviewer recommendation help developers? IEEE Transactions on Software Engineering (2018).

[24]

Bin Lin, Gregorio Robles, and Alexander Serebrenik. 2017. Developer turnover in global, industrial open source projects: Insights from applying survival analysis. In 2017 IEEE 12th International Conference on Global Software Engineering (ICGSE). IEEE, 66--75.

Digital Library

[25]

Jakub Lipcak and Bruno Rossi. 2018. A Large-Scale Study on Source Code Reviewer Recommendation. In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 378--387.

[26]

David W McDonald and Mark S Ackerman. 2000. Expertise recommender: a flexible recommendation system and architecture. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 231--240.

Digital Library

[27]

Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Software Engineering 21, 5 (2016), 2146--2189.

Digital Library

[28]

Ehsan Mirsaeedi and Peter C. Rigby. 2020. GitHub App: Sofia Bot. https://github.com/apps/sofiarec. (2020).

[29]

Ehsan Mirsaeedi and Peter C. Rigby. 2020. Replication Package and RelationalGit. https://github.com/cesel/relationalgit. (2020).

[30]

Ehsan Mirsaeedi and Peter C. Rigby. 2020. Sofia Bot Source code. https://github.com/cesel/Sofia. (2020).

[31]

Audris Mockus. 2009. Succession: Measuring transfer of code and developer productivity. In Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, 67--77.

Digital Library

[32]

Audris Mockus. 2010. Organizational volatility and its effects on software defects. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. ACM, 117--126.

Digital Library

[33]

Audris Mockus and James D Herbsleb. 2002. Expertise browser: a quantitative approach to identifying expertise. In Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. IEEE, 503--512.

[34]

Mathieu Nassif and Martin P Robillard. 2017. Revisiting turnover-induced knowledge loss in software projects. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 261--272.

[35]

Loo Geok Pee, Atreyi Kankanhalli, Gek Woo Tan, and GZ Tham. 2014. Mitigating the impact of member turnover in information systems development projects. IEEE Transactions on Engineering Management 61, 4 (2014), 702--716.

[36]

Nancy Pekala. 2001. Holding on to top talent. Journal of Property management 66, 5 (2001), 22--22.

[37]

Adam Porter, Harvey Siy, Audris Mockus, and Lawrence Votta. 1998. Understanding the sources of variation in software inspections. ACM Transactions on Software Engineering and Methodology (TOSEM) 7, 1 (1998), 41--79.

Digital Library

[38]

Foyzur Rahman and Premkumar Devanbu. 2011. Ownership, experience and defects: a fine-grained study of authorship. In Proceedings of the 33rd International Conference on Software Engineering. ACM, 491--500.

Digital Library

[39]

Mohammad Masudur Rahman, Chanchal K Roy, and Jason A Collins. 2016. Correct: code reviewer recommendation in github based on cross-project and technology experience. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 222--231.

Digital Library

[40]

Mehvish Rashid, Paul M Clarke, and Rory V O'Connor. 2017. Exploring knowledge loss in open source software (OSS) projects. In International conference on software process improvement and capability determination. Springer, 481--495.

[41]

Peter Rigby, Brendan Cleary, Frederic Painchaud, Margaret-Anne Storey, and Daniel German. 2012. Contemporary peer review in action: Lessons from open source development. IEEE software 29, 6 (2012), 56--61.

[42]

Peter C Rigby and Christian Bird. 2013. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 202--212.

Digital Library

[43]

Peter C Rigby, Daniel M German, Laura Cowen, and Margaret-Anne Storey. 2014. Peer review on open-source software projects: Parameters, statistical models, and theory. ACM Transactions on Software Engineering and Methodology (TOSEM) 23, 4 (2014), 35.

Digital Library

[44]

Peter C Rigby and Margaret-Anne Storey. 2011. Understanding broadcast based peer review on open source software projects. In 2011 33rd International Conference on Software Engineering (ICSE). IEEE, 541--550.

Digital Library

[45]

P. C. Rigby, Y. C. Zhu, S. M. Donadelli, and A. Mockus. 2016. Quantifying and Mitigating Turnover-Induced Knowledge Loss: Case Studies of Chrome and a Project at Avaya. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). 1006--1016.

Digital Library

[46]

Martin P Robillard, Mathieu Nassif, and Shane McIntosh. 2018. Threats of Aggregating Software Repository Data. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 508--518.

[47]

Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, 181--190.

Digital Library

[48]

Pratyush N Sharma, John Hulland, and Sherae Daniel. 2012. Examining turnover in open source software projects using logistic hierarchical linear modeling approach. In IFIP International Conference on Open Source Systems. Springer, 331--337.

[49]

Meaghan Stovel and Nick Bontis. 2002. Voluntary turnover: knowledge management-friend or foe? Journal of intellectual Capital 3, 3 (2002), 303--322.

[50]

Patanamon Thongtanunam, Shane McIntosh, Ahmed E Hassan, and Hajimu Iida. 2016. Revisiting code ownership and its relationship with software quality in the scope of modern code review. In Proceedings of the 38th international conference on software engineering. ACM, 1039--1050.

Digital Library

[51]

Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 2017. Review Participation in Modern Code Review: An Empirical Study of the Android, Qt, and OpenStack Projects. Empirical Software Engineering 22, 2 (2017), 768--817.

Digital Library

[52]

Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, and Ken-ichi Matsumoto. 2015. Who should review my code? a file location-based code-reviewer recommendation approach for modern code review. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 141--150.

[53]

Zeynep Ton and Robert S Huckman. 2008. Managing the impact of employee turnover on performance: The role of process conformance. Organization Science 19, 1 (2008), 56--68.

Digital Library

[54]

Xin Xia, David Lo, Xinyu Wang, and Xiaohu Yang. 2015. Who should review this change?: Putting text and file location analyses together for more accurate recommendations. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 261--270.

Digital Library

[55]

Yue Yu, Huaimin Wang, Gang Yin, and Tao Wang. 2016. Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment? Information and Software Technology 74 (2016), 204--218.

Digital Library

[56]

Motahareh Bahrami Zanjani, Huzefa Kagdi, and Christian Bird. 2016. Automatically Recommending Peer Reviewers in Modern Code Review. IEEE Trans. Softw. Eng. 42, 6 (June 2016), 530--543.

Digital Library

Cited By

Wang LZhou YZhuang HLi QCui DZhao YWang LFilkov VRay BZhou M(2024)Unity Is Strength: Collaborative LLM-Based Agents for Code Reviewer RecommendationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695291(2235-2239)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695291
Yang LXu JZhang HWu FLyu JLi YBacchelli AFilkov VRay BZhou M(2024)GPP: A Graph-Powered Prioritizer for Code Review RequestsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694990(104-116)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694990
Kazemi FLamothe MMcIntosh S(2024)Characterizing the Prevalence, Distribution, and Duration of Stale Reviewer RecommendationsIEEE Transactions on Software Engineering10.1109/TSE.2024.342236950:8(2096-2109)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3422369
Show More Cited By

Recommendations

Quantifying and mitigating turnover-induced knowledge loss: case studies of chrome and a project at avaya
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

The utility of source code, as of other knowledge artifacts, is predicated on the existence of individuals skilled enough to derive value by using or improving it. Developers leaving a software project deprive the project of the knowledge of the ...
The Effect of Lengthening Job Tenure on Managers' Organizational Commitment and Turnover

<P>Using the psychological contract literature as a framework, we examined upwardly mobile managers' reactions to lengthening job tenure brought about by slower organizational growth and greater competition among peers. We hypothesized that managers ...
Turnover and Turnaway of IT Workers: A Person-Environment Fit Perspective
SIGMIS-CPR '20: Proceedings of the 2020 Computers and People Research Conference
Especially in view of the increasing skill shortage, the retention of information technology (IT) workers in organizations remains one of the three most important IT management issues for practitioners [11]. Organizations are confronted with two relevant ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
575
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)4

Reflects downloads up to 17 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang LZhou YZhuang HLi QCui DZhao YWang LFilkov VRay BZhou M(2024)Unity Is Strength: Collaborative LLM-Based Agents for Code Reviewer RecommendationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695291(2235-2239)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695291
Yang LXu JZhang HWu FLyu JLi YBacchelli AFilkov VRay BZhou M(2024)GPP: A Graph-Powered Prioritizer for Code Review RequestsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694990(104-116)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694990
Kazemi FLamothe MMcIntosh S(2024)Characterizing the Prevalence, Distribution, and Duration of Stale Reviewer RecommendationsIEEE Transactions on Software Engineering10.1109/TSE.2024.342236950:8(2096-2109)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3422369
Hajari FMalmir SMirsaeedi ERigby P(2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: Apr-2024
https://doi.org/10.1109/TSE.2024.3366753
Rong GYu YZhang YZhang HShen HShao DKuang HWang MWei ZXu YWang J(2024)Distilling Quality Enhancing Comments From Code Reviews to Underpin Reviewer RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.335681950:7(1658-1674)Online publication date: Jul-2024
https://doi.org/10.1109/TSE.2024.3356819
Tufano RDabić OMastropaolo ACiniselli MBavota G(2024)Code Review Automation: Strengths and Weaknesses of the State of the ArtIEEE Transactions on Software Engineering10.1109/TSE.2023.334817250:2(338-353)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TSE.2023.3348172
Di Marco NCinelli MAlipour SQuattrociocchi W(2024)Users Volatility on Reddit and VoatIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.337931811:5(5871-5879)Online publication date: Oct-2024
https://doi.org/10.1109/TCSS.2024.3379318
Koana ULe QRahman SCarlson CChew FNayebi M(2024)Examining ownership models in software teamsEmpirical Software Engineering10.1007/s10664-024-10538-529:6Online publication date: 27-Sep-2024
https://doi.org/10.1007/s10664-024-10538-5
Yang LZhang HXu JLyu JZhou XShao DGao SBacchelli A(2024)A preliminary investigation on using multi-task learning to predict change performance in code reviewsEmpirical Software Engineering10.1007/s10664-024-10526-929:6Online publication date: 28-Sep-2024
https://doi.org/10.1007/s10664-024-10526-9
Liu JDeng AXie QYue G(2023)A Code Reviewer Recommendation Approach Based on Attentive Neighbor Embedding PropagationElectronics10.3390/electronics1209211312:9(2113)Online publication date: 5-May-2023
https://doi.org/10.3390/electronics12092113
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents