research-article

Knowledge Transfer in Modern Code Review

Authors:

Gabriele Bavota,

Giuseppe Scanniello,

Michele LanzaAuthors Info & Claims

ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

Pages 230 - 240

https://doi.org/10.1145/3387904.3389270

Published: 12 September 2020 Publication History

Abstract

Knowledge transfer is one of the main goals of modern code review, as shown by several studies that surveyed and interviewed developers. While knowledge transfer is a clear expectation of the code review process, there are no analytical studies using data mined from software repositories to assess the effectiveness of code review in "training" developers and improve their skills over time. We present a mining-based study investigating how and whether the code review process helps developers to improve their contributions to open source projects over time. We analyze 32,062 peer-reviewed pull requests (PRs) made across 4,981 GitHub repositories by 728 developers who created their GitHub account in 2015. We assume that PRs performed in the past by a developer D that have been subject to a code review process have "transferred knowledge" to D. Then, we verify if over time (i.e., when more and more reviewed PRs are made by D), the quality of the contributions made by D to open source projects increases (as assessed by proxies we defined, such as the acceptance of PRs, or the polarity of the sentiment in the review comments left for the submitted PRs). With the above measures, we were unable to capture the positive impact played by the code review process on the quality of developers' contributions. This might be due to several factors, including the choices we made in our experimental design.Additional investigations are needed to confirm or contradict such a negative result.

References

[1]

E. M. Awad and H. M. Ghaziri. 2009. Knowledge Management. International Edition.

[2]

A. Bacchelli and C. Bird. 2013. Expectations, Outcomes, and Challenges of Modern Code Review. In Proceedings of the 2013 International Conference on Software Engineering (ICSE '13). 712--721.

[3]

T. Baum, O. Liskin, K. Niklas, and K. Schneider. 2016. A Faceted Classification Scheme for Change-Based Industrial Code Review Processes. In Proceedings of the 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS' 16). 74--85.

[4]

G. Bavota and B. Russo. 2015. Four eyes are better than two: On the impact of code reviews on software quality. In Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME '15). 81--90. https://doi.org/10.1109/ICSM.2015.7332454

Digital Library

[5]

O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey. 2013. The influence of non-technical factors on code review. In Proceedings of the 20th Working Conference on Reverse Engineering (WCRE 2013). 122--131.

[6]

M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens. 2014. Modern Code Reviews in Open-source Projects: Which Problems Do They Fix?. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 202--211.

Digital Library

[7]

M. Bernhart and T. Grechenig. 2013. On the understanding of programs with continuous code reviews. In Proceedings of the 21st International Conference on Program Comprehension (ICPC '13 ). 192--198. https://doi.org/10.1109/ICPC.2013.6613847

[8]

M. Bernhart, A. Mauczka, and T. Grechenig. 2010. Adopting Code Reviews for Agile Software Development. In 2010 Agile Conference. 44--47. https://doi.org/10.1109/AGILE.2010.18

[9]

F. Bjørnson and T. Dingsøyr. 2008. Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used. Information and Software Technology 50, 11 (2008), 1055--1068.

Digital Library

[10]

A. Bosu, M. Greiler, and C. Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR '15). 146--156. https://doi.org/10.1109/MSR.2015.21

[11]

F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli. 2018. Sentiment Polarity Detection for Software Development. Empirical Software Engineering 23, 3 (June 2018), 1352--1382. https://doi.org/10.1007/s10664-017-9546-9

Digital Library

[12]

M. Caulo, B. Lin, G. Bavota, G. Scanniello, and M. Lanza. 2020. https://tinyurl.com/wn9my8f.

[13]

D.Chen, K. Stolee, and T. Menzies. 2019. Replication Can Improve Prior Results: A GitHub Study of Pull Request Acceptance. In Proceedings of the 27th International Conference on Program Comprehension (ICPC '19 ). 179--190. https://doi.org/10.1109/ICPC.2019.00037

[14]

W.J. Conover. 1999. Practical nonparametric statistics. (1999).

[15]

J. Czerwonka, M. Greiler, and J. Tilford. 2015. Code Reviews Do Not Find Bugs: How the Current Code Review Best Practice Slows Us Down. In Proceedings of the 37th International Conference on Software Engineering (ICSE '15). IEEE Press, 27--28.

[16]

G. Destefanis, M. Ortu, D. Bowes, M. Marchesi, and R. Tonelli. 2018. On Measuring Affects of Github Issues' Commenters. In Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering (SEmotion '18). Association for Computing Machinery, New York, NY, USA, 14--19. https://doi.org/10.1145/3194932.3194936

[17]

R. J. Grissom and J. J. Kim. 2005. Effect sizes for research: A broad practical approach. Mahwah, NJ: Earlbaum (2005).

[18]

S. Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65--70.

[19]

M. Islam and M. Zibran. 2018. SentiStrength-SE: Exploiting Domain Specificity for Improved Sentiment Analysis in Software Engineering Text. Journal of Systems and Software 145 (08 2018). https://doi.org/10.1016/j.jss.2018.08.030

[20]

C. F. Kemerer and M. C. Paulk. 2009. The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data. Software Engineering, IEEE Transactions on 35, 4 (2009), 534--550.

Digital Library

[21]

O. Kononenko, T. Rose, O. Baysal, M. Godfrey, D. Theisen, and B. de Water. 2018. Studying Pull Request Merges: A Case Study of Shopify's Active Merchant. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP '18). 124--133.

Digital Library

[22]

Z. Li, Y. Yu, G. Yin, T. Wang, and H.-M. Wang. 2017. What Are They Talking About? Analyzing Code Reviews in Pull-Based Development Model. Journal of Computer Science and Technology 32 (11 2017), 1060--1075. https://doi.org/10.1007/s11390-017-1783-2

[23]

B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto. 2018. Sentiment analysis for software engineering: how far can we go?. In Proceedings of the 40th International Conference on Software Engineering, (ICSE' 18). 94--104.

[24]

S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan. 2014. The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). 192--201.

Digital Library

[25]

R. Morales, S. McIntosh, and F. Khomh. 2015. Do Code Review Practices Impact Design Quality? A Case Study of the Qt, VTK, and ITK Projects. In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER '15). 171--180.

[26]

R. Morales, S. McIntosh, and F. Khomh. 2015. Do code review practices impact design quality? A case study of the Qt, VTK, and ITK projects. Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER '15) (04 2015), 171--180. https://doi.org/10.1109/SANER.2015.7081827

[27]

H. Motulsky. 2010. Intuitive biostatistics: a non-mathematical guide to statistical thinking. Oxford University Press. http://books.google.it/books?id=R477U5bAZs4C

[28]

Alessandro Murgia, Giulio Concas, Sandro Pinna, Roberto Tonelli, and Ivana Turnu. 2009. Empirical study of software quality evolution in open source projects using agile practices. In Proc. of the 1st International Symposium on Emerging Trends in Software Metrics. 11.

[29]

M. V. Mäntylä and C. Lassenius. 2009. What Types of Defects Are Really Discovered in Code Reviews? IEEE Transactions on Software Engineering 35, 3 (May 2009), 430--448. https://doi.org/10.1109/TSE.2008.71

Digital Library

[30]

M. Ortu, M. Marchesi, and R. Tonelli. 2019. Empirical Analysis of Affect of Merged Issues on GitHub. In Proceedings of the 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion '19). 46--48. https://doi.org/10.1109/SEmotion.2019.00017

[31]

P. Pooput and P. Muenchaisri. 2018. Finding Impact Factors for Rejection of Pull Requests on GitHub. In Proceedings of the 2018 VII International Conference on Network, Communication and Computing (ICNCC 2018). 70--76.

[32]

M. M. Rahman and C. Roy. 2014. An Insight into the Pull Request of GitHub. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR '14). 364--367. https://doi.org/10.1145/2597073.2597121

[33]

P. C. Rigby, D. M. German, and M.-A. Storey. 2008. Open Source Software Peer Review Practices: A Case Study of the Apache Server. In Proceedings of the 30th International Conference on Software Engineering (ICSE '08). 541--550.

Digital Library

[34]

P. C. Rigby and M.-A. Storey. 2011. Understanding Broadcast Based Peer Review on Open Source Software Projects. In Proceedings of the 33rd International Conference on Software Engineering (ICSE '11). 541--550.

Digital Library

[35]

M. Silva, M. Valente, and R. Terra. 2016. Does Technical Debt Lead to the Rejection of Pull Requests?. In Proceedings of the XII Brazilian Symposium on Information Systems on Brazilian Symposium on Information Systems: Information Systems in the Cloud Computing Era-Volume 1. 248--254.

[36]

D. Soares, M. de Lima Júnior, L. Murta, and A. Plastino. 2015. Acceptance Factors of Pull Requests in Open-Source Projects. In Proceedings of the 30th Annual ACM Symposium on Applied Computing (SAC '15). Association for Computing Machinery, New York, NY, USA, 1541--1546. https://doi.org/10.1145/2695664.2695856

[37]

D. M. Soares, M. L. d. L. Júnior, L. Murta, and A. Plastino. 2015. Rejection Factors of Pull Requests Filed by Core Team Developers in Software Projects with High Acceptance Rates. In Proceedings of the 14th International Conference on Machine Learning and Applications (ICMLA '15). 960--965. https://doi.org/10.1109/ICMLA.2015.41

[38]

Ramanath Subramanyam and Mayuram S. Krishnan. 2003. Empirical analysis of ck metrics for object-oriented design complexity: Implications for software defects. IEEE Transactions on software engineering 29, 4 (2003), 297--310.

Digital Library

[39]

J. Tsay, L. Dabbish, and J. Herbsleb. 2014. Influence of Social and Technical Factors for Evaluating Contribution in GitHub. In Proceedings of the 36th International Conference on Software Engineering (ICSE '14). Association for Computing Machinery, New York, NY, USA, 356--366. https://doi.org/10.1145/2568225.2568315

[40]

J. W. Tukey. 1977. Exploratory Data Analysis. Addison-Wesley.

[41]

P. Weißgerber, D. Neu, and S.Diehl. 2008. Small Patches Get in!. In Proceedings of the 2008 International Working Conference on Mining Software Repositories (MSR '08). 67--76.

Digital Library

[42]

A. Zeller. 2005. Why Programs Fail: A Guide to Systematic Debugging. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Digital Library

Cited By

Liang MCharoenwet WThongtanunam PSpinellis DConstantinou EBacchelli A(2024)Curated Email-Based Code Reviews DatasetsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644872(294-298)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644872
Rong GYu YZhang YZhang HShen HShao DKuang HWang MWei ZXu YWang J(2024)Distilling Quality Enhancing Comments From Code Reviews to Underpin Reviewer RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.335681950:7(1658-1674)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3356819
Yang LZhang HXu JLyu JZhou XShao DGao SBacchelli A(2024)A prelinary investigation on using multi-task learning to predict change performance in code reviewsEmpirical Software Engineering10.1007/s10664-024-10526-929:6Online publication date: 28-Sep-2024
https://doi.org/10.1007/s10664-024-10526-9
Show More Cited By

Index Terms

Knowledge Transfer in Modern Code Review
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Sentiment analysis
2. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
  2. Software notations and tools
    1. Software libraries and repositories

Recommendations

Code review quality: how developers see it
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

In a large, long-lived project, an effective code review process is key to ensuring the long-term quality of the code base. In this work, we study code review practices of a large, open source project, and we investigate how the developers themselves ...
Code smells detection via modern code review: a study of the OpenStack and Qt communities
Abstract
Code review plays an important role in software quality control. A typical review process involves a careful check of a piece of code in an attempt to detect and locate defects and other quality issues/violations. One type of issue that may impact ...
Demystifying code snippets in code reviews: a study of the OpenStack and Qt communities and a practitioner survey
Abstract
Code review is widely known as one of the best practices for software quality assurance in software development. In a typical code review process, reviewers check the code committed by developers to ensure the quality of the code, during which ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

July 2020

481 pages

ISBN:9781450379588

DOI:10.1145/3387904

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPC '20

Sponsor:

SIGSOFT

ICPC '20: 28th International Conference on Program Comprehension

July 13 - 15, 2020

Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
275
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)8

Reflects downloads up to 28 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liang MCharoenwet WThongtanunam PSpinellis DConstantinou EBacchelli A(2024)Curated Email-Based Code Reviews DatasetsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644872(294-298)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644872
Rong GYu YZhang YZhang HShen HShao DKuang HWang MWei ZXu YWang J(2024)Distilling Quality Enhancing Comments From Code Reviews to Underpin Reviewer RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.335681950:7(1658-1674)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3356819
Yang LZhang HXu JLyu JZhou XShao DGao SBacchelli A(2024)A prelinary investigation on using multi-task learning to predict change performance in code reviewsEmpirical Software Engineering10.1007/s10664-024-10526-929:6Online publication date: 28-Sep-2024
https://doi.org/10.1007/s10664-024-10526-9
Silva DPabasara WSangkavi SL.G.A.T.D WW.M.K.H WS.A R(2023)The Effectiveness of Code Reviews on Improving Software Quality: An Empirical StudyInternational Journal of Recent Technology and Engineering (IJRTE)10.35940/ijrte.B7666.071222312:2(1-10)Online publication date: 30-Jul-2023
https://doi.org/10.35940/ijrte.B7666.0712223
Yang LXu JZhang YZhang HBacchelli AChandra SBlincoe KTonella P(2023)EvaCRC: Evaluating Code Review CommentsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616245(275-287)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616245
Badampudi DUnterkalmsteiner MBritto R(2023)Modern Code Reviews—Survey of Literature and PracticeACM Transactions on Software Engineering and Methodology10.1145/358500432:4(1-61)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3585004
Baciejowski KGarbala DZmijewski SMadeyski L(2023)Are Code Review Smells and Metrics Useful in Pull Request-Level Software Defect Prediction?Developments in Information and Knowledge Management Systems for Business Applications10.1007/978-3-031-27506-7_2(27-52)Online publication date: 8-Apr-2023
https://doi.org/10.1007/978-3-031-27506-7_2
Yang LZhang HZhang FZhang XRong GHarman MMiller H(2022)An industrial experience report on retro-inspectionProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513055(43-52)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510457.3513055
Yang LZhang HZhang FZhang XRong G(2022)An Industrial Experience Report on Retro-Inspection2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)10.1109/ICSE-SEIP55303.2022.9793980(43-52)Online publication date: May-2022
https://doi.org/10.1109/ICSE-SEIP55303.2022.9793980
Doğan ETüzün E(2022)Towards a taxonomy of code review smellsInformation and Software Technology10.1016/j.infsof.2021.106737142:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.infsof.2021.106737
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents