research-article

The impact of test ownership and team structure on the reliability and effectiveness of quality test runs

Authors:

Nachiappan NagappanAuthors Info & Claims

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/2652524.2652535

Published: 18 September 2014 Publication History

Abstract

Context: Software testing is a crucial step in most software development processes. Testing software is a key component to manage and assess the risk of shipping quality products to customers. But testing is also an expensive process and changes to the system need to be tested thoroughly which may take time. Thus, the quality of a software product depends on the quality of its underlying testing process and on the effectiveness and reliability of individual test cases.

Goal: In this paper, we investigate the impact of the organizational structure of test owners on the reliability and effectiveness of the corresponding test cases. Prior empirical research on organizational structure has focused only on developer activity. We expand the scope of empirical knowledge by assessing the impact of organizational structure on testing activities.

Method: We performed an empirical study on the Windows build verification test suites (BVT) and relate effectiveness and reliability measures of each test run to the complexity and size of the organizational sub-structure that enclose all owners of test cases executed.

Results: Our results show, that organizational structure impacts both test effectiveness and test execution reliability. We are also able to predict effectiveness and reliability with fairly high precision and recall values.

Conclusion: We suggest to review test suites with respect to their organizational composition. As indicated by the results of this study, this would increase the effectiveness and reliability, development speed and developer satisfaction.

References

[1]

Bird, C., Nagappan, N., Murphy, B., Gall, H., and Devanbu, P. Don'T Touch My Code!: Examining the Effects of Ownership on Software Quality. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (2011), ACM, 4--14.

Digital Library

[2]

Nordberg, M. E., III. Managing code ownership. Software, IEEE, 20 (Mar 2003), 26--33.

Digital Library

[3]

Rahman, F. and Devanbu, P. Ownership, Experience and Defects: A Fine-grained Study of Authorship. In Proceedings of the 33rd International Conference on Software Engineering (2011), ACM, 491--500.

Digital Library

[4]

Bird, C. and Zimmermann, T. Assessing the Value of Branches with What-if Analysis. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (Cary, North Carolina, 2012), ACM, 45:1--45:11.

Digital Library

[5]

Nagappan, N., Murphy, B., and Basili, V. The Influence of Organizational Structure on Software Quality: An Empirical Case Study. In Proceedings of the 30th International Conference on Software Engineering (2008), ACM, 521--530.

Digital Library

[6]

Czerwonka, J., Nagappan, N., Schulte, W., and Murphy, B. CODEMINE: Building a Software Development Data Analytics Platform at Microsoft. Software, IEEE, 30, 4 (2013), 64--71.

Digital Library

[7]

Witten, I. H. and Frank, E. Data mining: practical machine learning tools and techniques with Java implementations. SIGMOD Rec., 31 (mar 2002), 76--77.

Digital Library

[8]

Team, R. D. C. R: A Language and Environment for Statistical Computing., 2010. R Foundation for Statistical Computing.

[9]

Kuhn, M. caret: Classification and Regression Training., 2011.

[10]

Weyuker, E. J., Ostrand, T. J., and Bell, R. M. Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models. Empirical Software Engineering, 13 (2008), 539--559.

Digital Library

[11]

Meneely, A. and Williams, L. Secure Open Source Collaboration: An Empirical Study of Linus' Law. In Proceedings of the 16th ACM Conference on Computer and Communications Security (2009), ACM, 453--462.

Digital Library

[12]

Curtis, B., Krasner, H., and Iscoe, N. A Field Study of the Software Design Process for Large Systems. Commun. ACM, 31 (nov 1988), 1268--1287.

Digital Library

[13]

Robillard, P. N. The Role of Knowledge in Software Development. Commun. ACM, 42 (jan 1999), 87--92.

Digital Library

[14]

Mockus, A. and Weiss, D. M. Predicting risk of software changes. Bell Labs Technical Journal, 5 (2000), 169--180.

[15]

Mockus, A. Organizational Volatility and Its Effects on Software Defects. In Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering (2010), ACM, 117--126.

Digital Library

[16]

Karus, S. and Dumas, M. Code Churn Estimation Using Organisational and Code Metrics: An Experimental Comparison. Inf. Softw. Technol., 54 (feb 2012), 203--211.

Digital Library

[17]

Bird, C., Nagappan, N., Devanbu, P., Gall, H., and Murphy, B. Does Distributed Development Affect Software Quality? An Empirical Case Study of Windows Vista. In Proceedings of the 31st International Conference on Software Engineering (2009), IEEE Computer Society, 518--528.

Digital Library

[18]

Cataldo, M. and Ehrlich, K. The Impact of the Structure of Communication Patterns in Global Software Development: An Empirical Analysis of a Project Using Agile Methods. Institute for Software Research, Carnegie Mellon University (2011).

[19]

Basili, V. R. and Caldiera, G. Improve Soft-ware Quality by Reusing Knowledge and Experience. Sloan management review, 37 (1995), 55--55.

[20]

Tamburri, D. A., Lago, P., and Vliet, H.v. Organizational Social Structures for Software Engineering. ACM Comput. Surv., 46 (jul 2013), 3:1--3:35.

Digital Library

[21]

Bettenburg, N. and Hassan, A. E. Studying the Impact of Social Interactions on Software Quality. Empirical Softw. Engg., 18 (apr 2013), 375--431.

Digital Library

[22]

Weyuker, E. J. Can we measure software testing effectiveness? In Software Metrics Symposium, 1993. Proceedings., First International (May 1993), 100--107.

[23]

Basili, V. R. and Selby, R. W. Comparing the Effectiveness of Software Testing Strategies. IEEE Trans. Softw. Eng., 13, 12 (December 1987), 1278--1296.

Digital Library

[24]

Goradia, T. Dynamic impact analysis: a cost-effective technique to enforce error-propagation. In Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis (Cambridge, Massachusetts, USA, 1993), ACM, 171--181.

Digital Library

[25]

Zhang, Y., Zhao, X., Zhang, X., and Zhang, T. Test effectiveness index: Integrating product metrics with process metrics. In Cyber Technology in Automation, Control, and Intelligent Systems (CYBER), 2012 IEEE International Conference on (May 2012), 54--57.

[26]

Whyte, G. and Mulder, D. L. Mitigating the Impact of Software Test Constraints on Software Testing Effectiveness. Electronic Journal of Information Systems Evaluation, 14 (2011), 254--270.

[27]

Rothermel, G., Untch, R. H., Chu, C., and Harrold, M. J. Test case prioritization: an empirical study. In Software Maintenance, 1999. (ICSM '99) Proceedings. IEEE International Conference on (1999), 179--188.

Digital Library

[28]

Elbaum, S., Malishevsky, A. G., and Rothermel, G. Test case prioritization: a family of empirical studies. Software Engineering, IEEE Transactions on, 28 (Feb 2002), 159--182.

Digital Library

[29]

Braione, P., Denaro, G., Mattavelli, A., Vivanti, M., and Muhammad, A. An industrial case study of the effectiveness of test generators. In Automation of Software Test (AST), 2012 7th International Workshop on (June 2012), 50--56.

Digital Library

[30]

Brennan, R. L. Generalizability theory. Educational Measurement: Issues and Practice, 11 (1992), 27--34.

[31]

Shavelson, R. J. and Webb, N. M. Generalizability theory: A primer. Sage, 1991.

[32]

Urbano, J., Marrero, M., and Mart\'\in, D. On the Measurement of Test Collection Reliability. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (2013), ACM, 393--402.

Digital Library

Cited By

Koana ULe QRahman SCarlson CChew FNayebi M(2024)Examining ownership models in software teamsEmpirical Software Engineering10.1007/s10664-024-10538-529:6Online publication date: 27-Sep-2024
https://doi.org/10.1007/s10664-024-10538-5
Koana UChew FCarlson CNayebi MChandra SBlincoe KTonella P(2023)Ownership in the Hands of Accountability at Brightsquid: A Case Study and a Developer SurveyProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613890(2008-2019)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613890
Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Show More Cited By

Index Terms

The impact of test ownership and team structure on the reliability and effectiveness of quality test runs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Process validation

Recommendations

A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2014

461 pages

ISBN:9781450327749

DOI:10.1145/2652524

General Chair:
Maurizio Morisio
Politecnico di Torino, Italy
,
Program Chairs:
Tore Dybå
Sintef, Norway
,
Marco Torchiano
Politecnico di Torino, Italy

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '14

Sponsor:

SIGSOFT

ESEM '14: 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 18 - 19, 2014

Torino, Italy

Acceptance Rates

ESEM '14 Paper Acceptance Rate 23 of 123 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
268
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Koana ULe QRahman SCarlson CChew FNayebi M(2024)Examining ownership models in software teamsEmpirical Software Engineering10.1007/s10664-024-10538-529:6Online publication date: 27-Sep-2024
https://doi.org/10.1007/s10664-024-10538-5
Koana UChew FCarlson CNayebi MChandra SBlincoe KTonella P(2023)Ownership in the Hands of Accountability at Brightsquid: A Case Study and a Developer SurveyProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613890(2008-2019)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613890
Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Cannavacciuolo CMariani L(2022)Smoke Testing of Cloud Systems2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00016(47-57)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00016
Suri VRangaswamy NJoshi TJoshi MNanavati SChandwani RSingh PKumar NVeeraraghavan R(2019)Tool smiths in off-shored workProceedings of the Tenth International Conference on Information and Communication Technologies and Development10.1145/3287098.3287112(1-10)Online publication date: 4-Jan-2019
https://dl.acm.org/doi/10.1145/3287098.3287112
Syed-Mohamad SMd. Akhir N(2019)SoReady: An Extension of the Test and Defect Coverage-Based Analytics Model for Pull-Based Software Development2019 26th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC48747.2019.00011(9-14)Online publication date: Dec-2019
https://doi.org/10.1109/APSEC48747.2019.00011
Jiang HLi XYang ZXuan JUchitel SOrso ARobillard M(2017)What causes my test alarm?Proceedings of the 39th International Conference on Software Engineering10.1109/ICSE.2017.71(712-723)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE.2017.71
Just SHerzig KCzerwonka JMurphy B(2016)Switching to Git: The Good, the Bad, and the Ugly2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2016.38(400-411)Online publication date: Oct-2016
https://doi.org/10.1109/ISSRE.2016.38
Jiang YAdams BPenta MPinzger MRobbes R(2015)Co-evolution of infrastructure and source codeProceedings of the 12th Working Conference on Mining Software Repositories10.5555/2820518.2820527(45-55)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2820518.2820527
Greiler MHerzig KCzerwonka JPenta MPinzger MRobbes R(2015)Code ownership and software qualityProceedings of the 12th Working Conference on Mining Software Repositories10.5555/2820518.2820522(2-12)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2820518.2820522
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents