article

On Building Prediction Systems for Software Engineers

Authors:

Martin Shepperd,

Michelle Cartwright,

Gada KadodaAuthors Info & Claims

Empirical Software Engineering, Volume 5, Issue 3

Pages 175 - 182

https://doi.org/10.1023/A:1026582314146

Published: 01 November 2000 Publication History

Abstract

Building and evaluating prediction systems is an important activity for software engineering researchers. Increasing numbers of techniques and datasets are now being made available. Unfortunately systematic comparison is hindered by the use of different accuracy indicators and evaluation processes. We argue that these indicators are statistics that describe properties of the estimation errors or residuals and that the sensible choice of indicator is largely governed by the goals of the estimator. For this reason it may be helpful for researchers to provide a range of indicators. We also argue that it is useful to formally test for significant differences between competing prediction systems and note that where only a few cases are available this can be problematic, in other words the research instrument may have insufficient power. We demonstrate that this is the case for a well known empirical study of cost models. Simulation, however, could be one means of overcoming this difficulty.

References

[1]

Benington, H. D. 1956. Production of large computer programs. Proc. Symp. on Advanced Computer Programs for Digital Computers. Washington, D.C.: Office of Naval Research.

[2]

Boehm, B. W. 1984. Software engineering economics. IEEE Trans. on Softw. Eng. 10(1): 4-21.

Digital Library

[3]

Conte, S., Dunsmore, H., and Shen, V. Y. 1986. Software Engineering Metrics and Models. Menlo Park, CA: Benjamin Cummings.

[4]

Desharnais, J. M. 1989. Analyse statistique de la productivitie des projects informatique a partie de la technique des point des fonction. University of Montreal.

[5]

Finnie, G. R., Wittig, G. E., and Desharnais, J.-M. 1997. A comparison of software effort estimation techniques using function points with neural networks, case based reasoning and regression models. J. of Systems Software 39: 281-289.

Digital Library

[6]

Jeffery, D. R., and Low, G. C. 1990. Calibrating estimation tools for software development. Softw. Eng. J. 5(4): 215-221.

Digital Library

[7]

Kemerer, C. F. 1987. An empirical validation of software cost estimation models. CACM 30(5): 416-429.

Digital Library

[8]

Kitchenham, B. A., and Linkman, S. G. 1997. Estimates, uncertainty and risk. IEEE Softw. 14(3): 69-74.

Digital Library

[9]

Kitchenham, B. A., and Taylor, N. R. 1984. Software cost models. ICL Tech. J. 4(3): 73-102.

[10]

Kok, P., Kitchenham, B. A., and Kirakowski, J. 1990. The MERMAID approach to software cost estimation. Proc. Esprit Technical Week.

[11]

Lo, B. W. N., and Gao, X. Assessing software cost estimation models: Criteria for accuracy, consistency and regression. Australian J. of Information Systems 5(1): 30-44.

[12]

Pickard, L., Kitchenham, B., and Linkman, S. 1999. An investigation of analysis techniques for software datasets. Technical Report No. TR99-05, Keele University.

[13]

Samson, B., Ellison, D., and Dugard, P. 1997. Software cost estimation using an Albus Perceptron (CMAC). Information & Softw. Technol. 39(1-2).

[14]

Shepperd, M. J., and Schofield, C. 1997. Estimating software project effort using analogies. IEEE Trans. on Softw. Eng. 23(11): 736-743.

Digital Library

[15]

Stensrud, E., and Myrtveit, I. 1998. Human performance estimating with analogy and regression models: An empirical validation. Proc. 5th Intl. Metrics Symp. Bethesda, MD: IEEE Computer Society.

[16]

Walston, C. E., and Felix, C. P. 1997. A method of programming measurement and estimation. IBM Syst. J. 16(1): 54-73.

Digital Library

Cited By

Chowdhury SUddin GHemmati HHolmes R(2024)Method-level Bug Prediction: Problems and PromisesACM Transactions on Software Engineering and Methodology10.1145/364033133:4(1-31)Online publication date: 13-Jan-2024
https://dl.acm.org/doi/10.1145/3640331
Xia TFu WShu RAgrawal RMenzies T(2022)Predicting health indicators for open source projects (using hyperparameter optimization)Empirical Software Engineering10.1007/s10664-022-10171-027:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10171-0
Miranda PFaria JCorreia FFares AGraça RMoreira JHung CHong JBechini ASong E(2021)An analysis of Monte Carlo simulations for forecasting software projectsProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3442030(1550-1558)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3442030
Show More Cited By

Recommendations

Evaluating prediction systems in software project estimation

Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to ...
Cross versus Within-Company Cost Estimation Studies: A Systematic Review

The objective of this paper is to determine under what circumstances individual organizations would be able to rely on cross-company-based estimation models. We performed a systematic review of studies that compared predictions from cross-company models ...
An extended study on applicability and performance of homogeneous cross-project defect prediction approaches under homogeneous cross-company effort estimation situation
Abstract
Software effort estimation (SEE) models have been studied for decades. One of serious but typical situations for data-oriented models is the availability of datasets for training models. Cross-company software effort estimation (CCSEE) is a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 5, Issue 3

November 2000

126 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2000 Kluwer Academic Publishers.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 November 2000

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chowdhury SUddin GHemmati HHolmes R(2024)Method-level Bug Prediction: Problems and PromisesACM Transactions on Software Engineering and Methodology10.1145/364033133:4(1-31)Online publication date: 13-Jan-2024
https://dl.acm.org/doi/10.1145/3640331
Xia TFu WShu RAgrawal RMenzies T(2022)Predicting health indicators for open source projects (using hyperparameter optimization)Empirical Software Engineering10.1007/s10664-022-10171-027:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10171-0
Miranda PFaria JCorreia FFares AGraça RMoreira JHung CHong JBechini ASong E(2021)An analysis of Monte Carlo simulations for forecasting software projectsProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3442030(1550-1558)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3442030
Chowdhury SBorle SRomansky SHindle A(2019)GreenScalerEmpirical Software Engineering10.1007/s10664-018-9640-724:4(1649-1692)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10664-018-9640-7
Sarro FPetrozziello A(2018)Linear Programming as a Baseline for Software Effort EstimationACM Transactions on Software Engineering and Methodology10.1145/323494027:3(1-28)Online publication date: 17-Sep-2018
https://dl.acm.org/doi/10.1145/3234940
Sarro FPetrozziello AHarman MDillon LVisser WWilliams L(2016)Multi-objective software effort estimationProceedings of the 38th International Conference on Software Engineering10.1145/2884781.2884830(619-630)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2884781.2884830
Zhang WHuang LNg VGe J(2015)SMPLearnerAutomated Software Engineering10.1007/s10515-014-0161-322:1(111-141)Online publication date: 1-Mar-2015
https://dl.acm.org/doi/10.1007/s10515-014-0161-3
Afzal WTorkar RFeldt RGorschek T(2014)Prediction of faults-slip-through in large software projectsSoftware Quality Journal10.1007/s11219-013-9205-322:1(51-86)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.1007/s11219-013-9205-3
Yu L(2012)An evolutionary programming based asymmetric weighted least squares support vector machine ensemble learning methodology for software repository miningInformation Sciences: an International Journal10.1016/j.ins.2011.09.034191(31-46)Online publication date: 1-May-2012
https://dl.acm.org/doi/10.1016/j.ins.2011.09.034
Afzal WTorkar R(2011)ReviewExpert Systems with Applications: An International Journal10.1016/j.eswa.2011.03.04138:9(11984-11997)Online publication date: 1-Sep-2011
https://dl.acm.org/doi/10.1016/j.eswa.2011.03.041
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents