research-article

WhoDo: automating reviewer suggestions at scale

Authors:

Ranjita Bhagwan,

Christian Bird,

Chandra Maddila,

B. AshokAuthors Info & Claims

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 937 - 945

https://doi.org/10.1145/3338906.3340449

Published: 12 August 2019 Publication History

Abstract

Today's software development is distributed and involves continuous changes for new features and yet, their development cycle has to be fast and agile. An important component of enabling this agility is selecting the right reviewers for every code-change - the smallest unit of the development cycle. Modern tool-based code review is proven to be an effective way to achieve appropriate code review of software changes. However, the selection of reviewers in these code review systems is at best manual. As software and teams scale, this poses the challenge of selecting the right reviewers, which in turn determines software quality over time. While previous work has suggested automatic approaches to code reviewer recommendations, it has been limited to retrospective analysis. We not only deploy a reviewer suggestions algorithm - WhoDo - and evaluate its effect but also incorporate load balancing as part of it to address one of its major shortcomings: of recommending experienced developers very frequently. We evaluate the effect of this hybrid recommendation + load balancing system on five repositories within Microsoft. Our results are based around various aspects of a commit and how code review affects that. We attempt to quantitatively answer questions which are supposed to play a vital role in effective code review through our data and substantiate it through qualitative feedback of partner repositories.

References

[1]

A. Frank Ackerman, Lynne S. Buchwald, and Frank H. Lewski. 1989. Software Inspections: An Effective Verification Process. IEEE Softw. 6, 3 (May 1989), 31–36.

Digital Library

[2]

A. Frank Ackerman, Priscilla J. Fowler, and Robert G. Ebenau. 1984. Software Inspections and the Industrial Production of Software. In Proc. Of a Symposium on Software Validation: Inspection-testing-verification-alternatives. Elsevier North-Holland, Inc., New York, NY, USA, 13–40. http://dl.acm.org/citation.cfm?id= 3541.3543

Digital Library

[3]

Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013, David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, 712–721.

Digital Library

[4]

Vipin Balachandran. 2013. Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 931–940.

Digital Library

[5]

Olga Baysal, Oleksii Kononenko, Reid Holmes, and Michael W. Godfrey. 2012. The Secret Life of Patches: A Firefox Case Study. In 19th Working Conference on Reverse Engineering, WCRE 2012, Kingston, ON, Canada, October 15-18, 2012. IEEE Computer Society, 447–455.

Digital Library

[6]

Olga Baysal, Oleksii Kononenko, Reid Holmes, and Michael W. Godfrey. 2013. The influence of non-technical factors on code review. In 20th Working Conference on Reverse Engineering, WCRE 2013, Koblenz, Germany, October 14-17, 2013, Ralf Lämmel, Rocco Oliveto, and Romain Robbes (Eds.). IEEE Computer Society, 122– 131.

[7]

Amiangshu Bosu, Jeffrey C. Carver, Christian Bird, Jonathan D. Orbeck, and Christopher Chockley. 2017. Process Aspects and Social Dynamics of Contemporary Code Review: Insights from Open Source Development and Industrial Practice at Microsoft. IEEE Trans. Software Eng. 43, 1 (2017), 56–75.

Digital Library

[8]

Christoph Hannebauer, Michael Patalas, Sebastian Stünkel, and Volker Gruhn. 2016. Automatically recommending code reviewers based on their expertise: An empirical comparison. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 99–110.

Digital Library

[9]

Jonathan L Herlocker, Joseph A Konstan, Loren G Terveen, and John T Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 5–53.

Digital Library

[10]

Gaeul Jeong, Sunghun Kim, Thomas Zimmermann, and Kwangkeun Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. Research on software analysis for error-free computing center Tech-Memo (ROSAEC MEMO 2009-006) (2009), 1–18.

[11]

Yujuan Jiang, Bram Adams, and Daniel M. Germán. 2013. Will my patch make it? and how fast?: case study on the Linux kernel. In Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, San Francisco, CA, USA, May 18-19, 2013, Thomas Zimmermann, Massimiliano Di Penta, and Sunghun Kim (Eds.). IEEE Computer Society, 101–110.

Digital Library

[12]

6624016

[13]

Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W Godfrey. 2015. Investigating code review quality: Do people and participation matter?. In 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 111–120.

Digital Library

[14]

Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2014. The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects. In 11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 - June 1, 2014, Hyderabad, India, Premkumar T. Devanbu, Sung Kim, and Martin Pinzger (Eds.). ACM, 192–201.

Digital Library

[15]

Ali Ouni, Raula Gaikovina Kula, and Katsuro Inoue. 2016. Search-based peer reviewers recommendation in modern code review. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 367–377.

[16]

Mohammad Masudur Rahman, Chanchal K Roy, and Jason A Collins. 2016. Correct: code reviewer recommendation in github based on cross-project and technology experience. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 222–231.

Digital Library

[17]

M. M. Rahman, C. K. Roy, J Redl, and J. Collins. 2016. CORRECT: Code Reviewer Recommendation at GitHub for Vendasta Technologies. In Proc. ASE. 792–797.

Digital Library

[18]

Peter C. Rigby and Christian Bird. 2013. Convergent contemporary software peer review practices. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13, Saint Petersburg, Russian Federation, August 18-26, 2013, Bertrand Meyer, Luciano Baresi, and Mira Mezini (Eds.). ACM, 202–212.

Digital Library

[19]

Peter C. Rigby, Brendan Cleary, Frédéric Painchaud, Margaret-Anne D. Storey, and Daniel M. Germán. 2012. Contemporary Peer Review in Action: Lessons from Open Source Development. IEEE Software 29, 6 (2012), 56–61.

Digital Library

[20]

Peter C. Rigby, Daniel M. Germán, and Margaret-Anne D. Storey. 2008. Open source software peer review practices: a case study of the apache server. In 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008, Wilhelm Schäfer, Matthew B. Dwyer, and Volker Gruhn (Eds.). ACM, 541–550.

Digital Library

[21]

Peter C. Rigby and Margaret-Anne D. Storey. 2011. Understanding broadcast based peer review on open source software projects. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21-28, 2011, Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.). ACM, 541–550.

Digital Library

[22]

Patanamon Thongtanunam, Raula Gaikovina Kula, Ana Erika Camargo Cruz, Norihiro Yoshida, and Hajimu Iida. 2014. Improving code review effectiveness through reviewer recommendations. In Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering. ACM, 119–122.

Digital Library

[23]

Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, and Ken-ichi Matsumoto. 2015. Who should review my code? A file location-based code-reviewer recommendation approach for modern code review. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 141–150.

[24]

Peter Weißgerber, Daniel Neu, and Stephan Diehl. 2008. Small patches get in!. In Proceedings of the 2008 International Working Conference on Mining Software Repositories, MSR 2008 (Co-located with ICSE), Leipzig, Germany, May 10-11, 2008, Proceedings, Ahmed E. Hassan, Michele Lanza, and Michael W. Godfrey (Eds.). ACM, 67–76.

Digital Library

[25]

Xin Xia, David Lo, Xinyu Wang, and Xiaohu Yang. 2015. Who should review this change?: Putting text and file location analyses together for more accurate recommendations. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 261–270.

Digital Library

[26]

Xin Yang, Norihiro Yoshida, Raula Gaikovina Kula, and Hajimu Iida. 2016. Peer review social network (PeRSoN) in open source projects. IEICE TRANSACTIONS on Information and Systems 99, 3 (2016), 661–670.

Cited By

Wang LZhou YZhuang HLi QCui DZhao YWang LFilkov VRay BZhou M(2024)Unity Is Strength: Collaborative LLM-Based Agents for Code Reviewer RecommendationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695291(2235-2239)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695291
Hajari FMalmir SMirsaeedi ERigby P(2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: Apr-2024
https://doi.org/10.1109/TSE.2024.3366753
Rong GYu YZhang YZhang HShen HShao DKuang HWang MWei ZXu YWang J(2024)Distilling Quality Enhancing Comments From Code Reviews to Underpin Reviewer RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.335681950:7(1658-1674)Online publication date: Jul-2024
https://doi.org/10.1109/TSE.2024.3356819
Show More Cited By

Index Terms

WhoDo: automating reviewer suggestions at scale
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing
2. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
      1. Programming teams
  2. Software notations and tools
    1. Software configuration management and version control systems
    2. Software maintenance tools

Recommendations

Using a trust network to improve top-N recommendation
RecSys '09: Proceedings of the third ACM conference on Recommender systems

Top-N item recommendation is one of the important tasks of recommenders. Collaborative filtering is the most popular approach to building recommender systems which can predict ratings for a given user and item. Collaborative filtering can be extended ...
User Similarity Adjustment for Improved Recommendations
MIKE 2015: Proceedings of the Third International Conference on Mining Intelligence and Knowledge Exploration - Volume 9468

Recommender systems are becoming more and more attractive in both research and commercial communities due to Information overload problem and the popularity of the Internet applications. Collaborative Filtering, a popular branch of recommendation ...
Confer: A Conference Recommendation and Meetup Tool
CSCW '16 Companion: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion

One of the primary goals of academic conferences is to promote scientific exchange of advances among people who may otherwise not have the opportunity to hear from one another. We present Confer, a tool designed to help conference attendees find ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2019

1264 pages

ISBN:9781450355728

DOI:10.1145/3338906

General Chairs:
Marlon Dumas
University of Tartu, Estonia
,
Dietmar Pfahl
University of Tartu, Estonia
,
Program Chairs:
Sven Apel
Saarland University, Germany
,
Alessandra Russo
Imperial College, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 30, 2019

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
532
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)4

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang LZhou YZhuang HLi QCui DZhao YWang LFilkov VRay BZhou M(2024)Unity Is Strength: Collaborative LLM-Based Agents for Code Reviewer RecommendationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695291(2235-2239)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695291
Hajari FMalmir SMirsaeedi ERigby P(2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: Apr-2024
https://doi.org/10.1109/TSE.2024.3366753
Rong GYu YZhang YZhang HShen HShao DKuang HWang MWei ZXu YWang J(2024)Distilling Quality Enhancing Comments From Code Reviews to Underpin Reviewer RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.335681950:7(1658-1674)Online publication date: Jul-2024
https://doi.org/10.1109/TSE.2024.3356819
Tufano RDabić OMastropaolo ACiniselli MBavota G(2024)Code Review Automation: Strengths and Weaknesses of the State of the ArtIEEE Transactions on Software Engineering10.1109/TSE.2023.334817250:2(338-353)Online publication date: Feb-2024
https://doi.org/10.1109/TSE.2023.3348172
Qiao YWang JCheng CTang WLiang PZhao YLi B(2024)Code Reviewer Recommendation Based on a Hypergraph with Multiplex Relationships2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00049(417-428)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00049
Cury OAvelino GNeto PValente MBritto R(2024)Source code expert identification: Models and applicationInformation and Software Technology10.1016/j.infsof.2024.107445170(107445)Online publication date: Jun-2024
https://doi.org/10.1016/j.infsof.2024.107445
Cao ZLv SZhang XLi HMa QLi TGuo CGuo S(2024)Structuring Meaningful Code Review Automation in Developer CommunityEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106970127(106970)Online publication date: Jan-2024
https://doi.org/10.1016/j.engappai.2023.106970
Liu JDeng AXie QYue G(2023)A Code Reviewer Recommendation Approach Based on Attentive Neighbor Embedding PropagationElectronics10.3390/electronics1209211312:9(2113)Online publication date: 5-May-2023
https://doi.org/10.3390/electronics12092113
Badampudi DUnterkalmsteiner MBritto R(2023)Modern Code Reviews—Survey of Literature and PracticeACM Transactions on Software Engineering and Methodology10.1145/358500432:4(1-61)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3585004
Murphy-Hill EDicker JHorvath AHodges MEgelman CWeingart LJaspan CGreen CChen N(2023)Systemic Gender Inequities in Who Reviews CodeProceedings of the ACM on Human-Computer Interaction10.1145/35795277:CSCW1(1-59)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579527
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents