research-article

Cross versus Within-Company Cost Estimation Studies: A Systematic Review

Authors:

Barbara A. Kitchenham,

Guilherme H. TravassosAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 33, Issue 5

Pages 316 - 329

https://doi.org/10.1109/TSE.2007.1001

Published: 01 May 2007 Publication History

Abstract

The objective of this paper is to determine under what circumstances individual organizations would be able to rely on cross-company-based estimation models. We performed a systematic review of studies that compared predictions from cross-company models with predictions from within-company models based on analysis of project data. Ten papers compared cross-company and within-company estimation models; however, only seven presented independent results. Of those seven, three found that cross-company models were not significantly different from within-company models, and four found that cross-company models were significantly worse than within-company models. Experimental procedures used by the studies differed making it impossible to undertake formal meta-analysis of the results. The main trend distinguishing study results was that studies with small within-company data sets (i.e., <20 projects) that used leave-one-out cross validation all found that the within-company model was significantly different (better) from the cross-company model. The results of this review are inconclusive. It is clear that some organizations would be ill-served by cross-company models whereas others would benefit. Further studies are needed, but they must be independent (i.e., based on different data bases or at least different single company data sets) and should address specific hypotheses concerning the conditions that would favor cross-company or within-company models. In addition, experimenters need to standardize their experimental procedures to enable formal meta-analysis, and recommendations are made in Section 3.

References

[1]

B.W. Boehm, Software Engineering Economics. Prentice Hall, 1981.

Digital Library

[2]

L.C. Briand, K. El-Emam, K. Maxwell, D. Surmann, and I. Wieczorek, “An Assessment and Comparison of Common Cost Estimation Models,” Proc. 21st Int'l Conf. Software Eng., pp. 313-322, 1999.

Digital Library

[3]

L.C. Briand, T. Langley, and I. Wieczorek, “A Replicated Assessment of Common Software Cost Estimation Techniques,” Proc. 22nd Int'l Conf. Software Eng., pp. 377-386, 2000.

Digital Library

[4]

T. DeMarco, Controlling Software Projects: Management Measurement and Estimation. Yourdon Press, 1982.

Digital Library

[5]

T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit, “A Simulation Study of the Model Evaluation Criterion MMRE,” IEEE Trans. Software Eng., vol. 29, no. 11, pp. 985-995, Nov. 2003.

Digital Library

[6]

R. Jeffery, M. Ruhe, and I. Wieczorek, “A Comparative Study of Two Software Development Cost Modeling Techniques Using Multi-Organizational and Company-Specific Data,” Information and Software Technology, vol. 42, pp. 1009-1016, 2000.

[7]

R. Jeffery, M. Ruhe, and I. Wieczorek, “Using Public Domain Metrics to Estimate Software Development Effort,” Proc. Int'l Software Metrics Symp. (Metrics '01), pp. 16-27, 2001.

Digital Library

[8]

C.F. Kemerer, “An Empirical Validation of Software Cost Estimation Models,” Comm. ACM, vol. 30, no. 5, 1987.

Digital Library

[9]

C. Kirsopp and M. Shepperd, “Making Inferences with Small Numbers of Training Sets,” IEE Proc. Software, vol. 149, no. 5, pp.123-130, 2002.

[10]

B. Kitchenham, “Procedures for Performing Systematic Reviews,” joint technical report, Software Eng. Group, Keele Univ., and Empirical Software Eng., Nat'l ICT Australia, 2004.

[11]

B.A. Kitchenham and E. Mendes, “A Comparison of Cross-Company and Single-Company Effort Estimation Models for Web Applications,” Proc. Empirical Assessment inSoftware Eng. (EASE '04), pp.47-55, 2004.

[12]

B.A. Kitchenham and N.R. Taylor, “Software Cost Models,” ICL Technical J., pp. 73-102, 1984.

[13]

B.A. Kitchenham, E. Mendes, and G.H. Travassos, “A Systematic Review of Cross- versus Within-Company Cost Estimation Studies,” Proc. Empirical Assessment in Software Eng. (EASE' 06), http://ewic.bcs.org/conferences/2006/ease06/index.htm, 2006.

Digital Library

[14]

P.A.M. Kok, J. Kirakowski, and B.A. Kitchenham, “The MERMAID Approach to Software Cost Estimation,” Proc. European Strategic Programme for Research and Development in Information Technologies Technical Week (ESPRIT '90), pp. 296-314, 1990.

[15]

M. Lefley and M.J. Shepperd, “Using Genetic Programming to Improve Software Effort Estimation Based on General Data Sets,” Proc. Genetic and Evolutionary Computation Conf. (GECCO '03), pp.2477-2487, 2003.

Digital Library

[16]

M.W. Lipsey and D.B. Wilson, Practical Meta-Analysis. Sage, 2001.

[17]

C. Mair and M.J. Shepperd, “The Consistency of Empirical Comparisons of Regression and Analogy-Based Software Project Cost Prediction,” Proc. Int'l Symp. Empirical Software Eng. (ISESE '05), pp. 509-518, Nov. 2005.

[18]

K. Maxwell, L.V. Wassenhove, and S. Dutta, “Performance Evaluation of General and Company Specific Models in Software Development Effort Estimation,” Management Science, vol. 45, no. 6, pp. 77-83, June 1999.

Digital Library

[19]

E. Mendes and B.A. Kitchenham, “Further Comparison of Cross-Company and Within Company Effort Estimation Models for Web Applications,” Proc. Int'l Software Metrics Symp. (Metrics '04), pp.348-357, 2004.

Digital Library

[20]

E. Mendes, C. Lokan, R. Harrison, and C. Triggs, “A Replicated Comparison of Cross-Company and Within-Company Effort Estimation Models Using the ISBSG Database,” Proc. Int'l Software Metrics Symp. (Metrics '05), 2005.

Digital Library

[21]

F. Mosteller and J.W. Tukey, Data Analysis and Regression. Addison-Welsey, 1977.

[22]

M. Pai, M. McCulloch, J.D. Gorman, N. Pai, W. Enanoria, G. Kennedy, P. Tharyan, and J.M. ColfordJr., “Systematic Reviews and Meta-Analyses: An Illustrated Step-by-Step Guide,” Nat'l Medical J. India, vol. 17, no. 2, pp. 86-95, 2004.

[23]

M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell, 2006.

[24]

L.A. Putnam, “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE Trans. Software Eng., vol. 4, no. 4, 1978.

Digital Library

[25]

J. Stathis, “Early Lifecycle Specification Based on Sizing with Function Points,” BSc thesis, Univ. of New South Wales, 1993.

[26]

I. Wieczorek and M. Ruhe, “How Valuable Is Company-Specific Data Compared to Multi-Company Data for Software Cost Estimation?” Proc. Int'l Software Metrics Symp. (Metrics '02), pp.237-246, June 2002.

Digital Library

Cited By

Yigitbas EMazur J(2024)Augmented and Virtual Reality for Diet and Nutritional Education: A Systematic Literature ReviewProceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3652037.3652048(88-97)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1145/3652037.3652048
Namoun ATufail ANawaz WBenRhouma OAlshanqiti A(2023)A Systematic Literature Review on Service Composition for People with DisabilitiesComputational Intelligence and Neuroscience10.1155/2023/59345482023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/5934548
Juárez-Ramírez RNavarro CJiménez SRamírez ATapia-Ibarra VGuerra-García CPerez-Gonzalez HFernández-y-Fernández C(2023)A Taxonomic View of the Fundamental Concepts of Quantum Computing–A Software Engineering PerspectiveProgramming and Computing Software10.1134/S036176882308010849:8(682-704)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1134/S0361768823080108
Show More Cited By

Index Terms

Cross versus Within-Company Cost Estimation Studies: A Systematic Review
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Implementation management
        Pricing and resource allocation
      2. Software management
        Software maintenance

Recommendations

Cross- vs. within-company cost estimation studies revisited: an extended systematic review
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

[Objective] The objective of this paper is to extend a previously conducted systematic literature review (SLR) that investigated under what circumstances individual organizations would be able to rely on cross-company based estimation models. [Method] ...
A systematic review of cross- vs. within- company cost estimation studies
EASE'06: Proceedings of the 10th international conference on Evaluation and Assessment in Software Engineering

OBJECTIVE - The objective of this paper is to determine under what circumstances individual organisations would be able to rely on cross-company based estimation models.

METHOD - We performed a systematic review of studies that compared predictions from ...
Further Comparison of Cross-Company and Within-Company Effort Estimation Models for Web Applications
METRICS '04: Proceedings of the Software Metrics, 10th International Symposium

This paper extends a previous study, using data on 67 Web projects from the Tukutuku database, investigating to what extent a cross-company cost model can be successfully employed to estimate effort for projects that belong to a single company, where no ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 33, Issue 5

May 2007

93 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © 2007.

Publisher

IEEE Press

Publication History

Published: 01 May 2007

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

121
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yigitbas EMazur J(2024)Augmented and Virtual Reality for Diet and Nutritional Education: A Systematic Literature ReviewProceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3652037.3652048(88-97)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1145/3652037.3652048
Namoun ATufail ANawaz WBenRhouma OAlshanqiti A(2023)A Systematic Literature Review on Service Composition for People with DisabilitiesComputational Intelligence and Neuroscience10.1155/2023/59345482023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/5934548
Juárez-Ramírez RNavarro CJiménez SRamírez ATapia-Ibarra VGuerra-García CPerez-Gonzalez HFernández-y-Fernández C(2023)A Taxonomic View of the Fundamental Concepts of Quantum Computing–A Software Engineering PerspectiveProgramming and Computing Software10.1134/S036176882308010849:8(682-704)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1134/S0361768823080108
Kitchenham BMadeyski LBudgen D(2023)SEGRESS: Software Engineering Guidelines for REporting Secondary StudiesIEEE Transactions on Software Engineering10.1109/TSE.2022.317409249:3(1273-1298)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3174092
Zain ZSakri SIsmail N(2023)Application of Deep Learning in Software Defect PredictionInformation and Software Technology10.1016/j.infsof.2023.107175158:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107175
Moya APretel ENavarro EJaén J(2023)A systematic literature review of clustering techniques for patients with traumatic brain injuryArtificial Intelligence Review10.1007/s10462-023-10531-256:Suppl 1(351-419)Online publication date: 24-Jun-2023
https://dl.acm.org/doi/10.1007/s10462-023-10531-2
Azzeh MAlqasrawi YElsheikh Y(2023)A soft computing approach for software defect density predictionJournal of Software: Evolution and Process10.1002/smr.255336:4Online publication date: 14-Mar-2023
https://dl.acm.org/doi/10.1002/smr.2553
Alqasrawi YAzzeh MElsheikh Y(2022)Locally weighted regression with different kernel smoothers for software effort estimationScience of Computer Programming10.1016/j.scico.2021.102744214:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.scico.2021.102744
Uludağ ÖPhilipp PPutta APaasivaara MLassenius CMatthes F(2022)Revealing the state of the art of large-scale agile development researchJournal of Systems and Software10.1016/j.jss.2022.111473194:COnline publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111473
Shukla AKatt BNweke LYeng PWeldehawaryat G(2022)System security assuranceComputer Science Review10.1016/j.cosrev.2022.10049645:COnline publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1016/j.cosrev.2022.100496
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents