research-article

Using version control data to evaluate the impact of software tools: a case study of the version editor

Authors:

David L. Atkins,

Todd L. Graves,

Audris MockusAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 28, Issue 7

Pages 625 - 637

https://doi.org/10.1109/TSE.2002.1019478

Published: 01 July 2002 Publication History

Abstract

Software tools can improve the quality and maintainability of software, but are expensive to acquire, deploy, and maintain, especially in large organizations. We explore how to quantify the effects of a software tool once it has been deployed in a development environment. We present an effort-analysis method that derives tool usage statistics and developer actions from a project's change history (version control system) and uses a novel effort estimation algorithm to quantify the effort savings attributable to tool usage. We apply this method to assess the impact of a software tool called VE, a version-sensitive editor used in Bell Labs. VE aids software developers in coping with the rampant use of certain preprocessor directives (similar to #if/#endif in C source files). Our analysis found that developers were approximately 40 percent more productive when using VE than when using standard text editors.

References

[1]

{1} A.J. Albrecht and J.R, Gaffney, "Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation," IEEE Trans. Software Eng., vol. 9, no. 6, pp. 638-648, 1983.

[2]

{2} D. Atkins, T. Ball, T. Graves, and A. Mockus, "Using Version Control Data to Evaluate the Effectiveness of Software Tools," Proc. Int'l Conf. Software Eng., pp. 324-333, 1999.

Digital Library

[3]

{3} D. Atkins, A. Mockus, and H. Siy, "Measuring Technology Effects on Software Change Cost," Bell Labs Technical J., vol. 5, no. 2, pp. 7-18, Apr./June 2000.

[4]

{4} D.L. Atkins, "Version Sensitive Editing: Change History as a Programming Tool," Proc. Eighth Conf. Software Configuration Management (SCM-8), pp. 146-157, 1998.

Digital Library

[5]

{5} V. Basili and R. Reiter, "An Investigation of Human Factors in Software Development," Computer, vol. 12, no. 12, pp. 21-38, Dec. 1979.

Digital Library

[6]

{6} B. Boehm, Software Engineering Economics. Prentice Hall, 1981.

Digital Library

[7]

{7} A. Brown and K. Wallnau, "A Framework for Evaluating Software Technology," IEEE Software, pp. 39-49, Sept. 1996.

Digital Library

[8]

{8} T. Bruckhaus, N. Madhavji, I. Janssen, and J. Henshaw, "The Impact of Tools on Software Productivity," IEEE Software, pp. 29-38, Sept. 1996.

Digital Library

[9]

{9} J.O. Coplien, D.L. DeBruler, and M.B. Thompson, "The Delta System: A Nontraditional Approach to Software Version Management," Int'l Switching Symp., Mar. 1987.

[10]

{10} B. Curtis, "Substantiating Programmer Variability," Proc. IEEE 69, July 1981.

[11]

{11} A.P. Dempster, N. Laird, and D.B. Rubin, "Maximum Likelihood from Incomplete Data via the EM Algorithm," J. Royal Statistical soc. B, vol. 39, pp. 1-38, 1977.

[12]

{12} H. Gottfried and M. Burnett, "Programming Complex Objects in Spreadsheets: An Empirical Study Comparing Textual Formula Entry with Direct Manipulation and Gestures," Proc. Seventh Workshop Empirical Studies of Programming, 1997.

Digital Library

[13]

{13} T.L. Graves and A. Mockus, "Inferring Change Effort from Configuration Management Data," Metrics 98: Fifth Int'l Symp. Software Metrics, pp. 267-273, Nov. 1998.

Digital Library

[14]

{14} T.L. Graves and A. Mockus, "Identifying Productivity Drivers by Modeling Work Units Using Partial Data," Technometrics, Vol. 43, no. 2, pp. 168-179, May 2001.

[15]

{15} B. Kitchenham, L. Pickard, and S.L. Pfleeger, "Case Studies for Method and Tool Evaluation," IEEE Software, pp. 52-62, July 1995.

Digital Library

[16]

{16} A. Lawrence, A. Badre, and J. Stasko, "Empirically Evaluating the Use of Animations to Teach Algorithms," Proc. 1994 IEEE Symp. Visual Languages, pp. 48-54, Oct. 1994.

[17]

{17} J. Lyle and M. Weiser, "Experiments on Slicing-Based Debugging Tools," Proc. First Workshop Empirical Studies of Programming, June 1986.

Digital Library

[18]

{18} P. McCullagh and J.A. Nelder, Generalized Linear Models, second ed. New York: Chapman and Hall, 1989.

[19]

{19} A.K. Midha, "Software Configuration Management for the 21st Century," Bell Labs Technical J., vol. 2, no. 1, Winter 1997.

[20]

{20} A. Mockus and L.G. Votta, "Identifying Reasons for Software Changes Using Historic Databases," Int'l Conf. Software Maintenance, pp. 120-130, Oct. 2000.

Digital Library

[21]

{21} A. Mockus and D.M. Weiss, "Predicting Risk of Software Changes," Bell Labs Technical J., vol. 5, no. 2, pp. 169-180, Apr./ June 2000.

[22]

{22} V. Mosley, "How to Assess Tools Efficiently and Quantitatively," IEEE Software, pp. 29-32 May 1992.

Digital Library

[23]

{23} T. Omerod and L. Ball, "An Empirical Evaluation of TEd, a Techniques Editor for Prolog Programming," Proc. Sixth Workshop Empirical Studies of Programming, 1996.

[24]

{24} A. Pal and M. Thompson, "An Advanced Interface to a Switching Software Version Management System," Seventh Int'l Conf. Software Eng. for Telecomm. Switching Systems, July 1989.

[25]

{25} R. Poston and M. Sexton, "Evaluating and Selecting Testing Tools," IEEE Software, pp. 33-42, May 1992.

Digital Library

[26]

{26} M. Rochkind, "The Source Code Control System," IEEE Trans. Software Eng., vol. 1, no. 4, pp. 364-370, 1975.

Digital Library

[27]

{27} M. Shepperd and C. Schofield, "Estimating Software Project Effort Using Analogies," IEEE Trans. Software Eng., vol. 23, no. 12, pp. 736-743, Nov. 1997.

Digital Library

[28]

{28} B. Shneiderman, Designing the User Interface, second ed. Addison-Wesley, 1991.

Digital Library

[29]

{29} H. Siy and A. Mockus, "Measuring Domain Engineering Effects on Software Coding Cost," Metrics 99: Sixth Int'l Symp. Software Metrics, pp. 304-311, Nov. 1999.

Digital Library

[30]

{30} H. Siy and A. Mockus, "Measuring Domain Engineering Effects on Software Coding Cost," Metrics 99: Sixth Int'l Symp. Software Metrics, pp. 304-311, Nov. 1999.

Digital Library

[31]

{31} G. Snelting, "Reengineering of Configurations Based on Mathematical Concept Analysis," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 146-189, Apr. 1996.

Digital Library

[32]

{32} E.B. Swanson, "The Dimensions of Maintenance," Proc. Second Conf. Software Eng., pp. 492-497, 1976.

Digital Library

Cited By

Shaikh MVaast E(2023)Algorithmic Interactions in Open Source WorkInformation Systems Research10.1287/isre.2022.115334:2(744-765)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1287/isre.2022.1153
Mockus ARigby PAbreu RSuresh PChen YNagappan NChandra SBlincoe KTonella P(2023)Modeling the Centrality of Developer Output with Software Supply ChainsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613873(1809-1819)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613873
Lillack MKästner CBodden E(2018)Tracking Load-Time Configuration OptionsIEEE Transactions on Software Engineering10.1109/TSE.2017.275604844:12(1269-1291)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1109/TSE.2017.2756048
Show More Cited By

Index Terms

Using version control data to evaluate the impact of software tools: a case study of the version editor
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
      2. System management
        Quality assurance
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Software version control
  2. Software notations and tools
    1. Development frameworks and environments
    2. Software configuration management and version control systems

Recommendations

Using version control data to evaluate the impact of software tools
ICSE '99: Proceedings of the 21st international conference on Software engineering
Assessing the potentials of CASE-tools in software process improvement: a benchmarking study
SAST '96: Proceedings of the Proceedings of the Fourth International Symposium on Assessment of Software Tools (SAST '96)

CASE tools have been thought as one of the most important means for implementing the derived quality programs. Two basic questions should be answered to find the right CASE tool: what attributes the CASE tools should exhibit and how the existing tools ...
Using software metrics tools for maintenance decisions: a classroom exercise
SAST '96: Proceedings of the Proceedings of the Fourth International Symposium on Assessment of Software Tools (SAST '96)

We explore the use of software metrics tools to guide software maintenance decisions. A senior undergraduate class was given a copy of QUIPU, an implementation of the X.500 directory standard, and asked to determine which component of the system would be ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 28, Issue 7

July 2002

96 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © 2002.

Publisher

IEEE Press

Publication History

Published: 01 July 2002

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shaikh MVaast E(2023)Algorithmic Interactions in Open Source WorkInformation Systems Research10.1287/isre.2022.115334:2(744-765)Online publication date: 1-Jun-2023
Mockus ARigby PAbreu RSuresh PChen YNagappan NChandra SBlincoe KTonella P(2023)Modeling the Centrality of Developer Output with Software Supply ChainsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613873(1809-1819)Online publication date: 30-Nov-2023
Lillack MKästner CBodden E(2018)Tracking Load-Time Configuration OptionsIEEE Transactions on Software Engineering10.1109/TSE.2017.275604844:12(1269-1291)Online publication date: 1-Dec-2018
Linsbauer LBerger TGrünbacher P(2017)A classification of variation control systemsACM SIGPLAN Notices10.1145/3170492.313605452:12(49-62)Online publication date: 23-Oct-2017
Linsbauer LBerger TGrünbacher PFlatt MErdweg S(2017)A classification of variation control systemsProceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3136040.3136054(49-62)Online publication date: 23-Oct-2017
Behringer BPalz JBerger TUchitel SOrso ARobillard M(2017)PEoPLProceedings of the 39th International Conference on Software Engineering10.1109/ICSE.2017.58(563-574)Online publication date: 20-May-2017
Krüger JSchröter IKenner AKruczek CLeich TSeidl CTeixeira L(2016)FeatureCoPP: compositional annotationsProceedings of the 7th International Workshop on Feature-Oriented Software Development10.1145/3001867.3001876(74-84)Online publication date: 30-Oct-2016
Oliveira EConte TCristo MMendes EGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Software Project Managers' Perceptions of Productivity FactorsProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962626(1-6)Online publication date: 8-Sep-2016
Seidl CTeixeira L(2016)Proceedings of the 7th International Workshop on Feature-Oriented Software DevelopmentundefinedOnline publication date: 30-Oct-2016
Ko ALaToza TBurnett M(2015)A practical guide to controlled experiments of software engineering tools with human participantsEmpirical Software Engineering10.1007/s10664-013-9279-320:1(110-141)Online publication date: 1-Feb-2015
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents