Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/2499393.2499396acmotherconferencesArticle/Chapter ViewAbstractPublication PagespromiseConference Proceedingsconference-collections
research-article
Open access

An analysis of multi-objective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation

Published: 09 October 2013 Publication History

Abstract

Background: Previous work showed that Multi-objective Evolutionary Algorithms (MOEAs) can be used for training ensembles of learning machines for Software Effort Estimation (SEE) by optimising different performance measures concurrently. Optimisation based on three measures (LSD, MMRE and PRED(25)) was analysed and led to promising results in terms of performance on these and other measures.
Aims: (a) It is not known how well ensembles trained on other measures would behave for SEE, and whether training on certain measures would improve performance particularly on these measures. (b) It is also not known whether it is best to include all SEE models created by the MOEA into the ensemble, or solely the models with the best training performance in terms of each measure being optimised. Investigating (a) and (b) is the aim of this work.
Method: MOEAs were used to train ensembles by optimising four different sets of performance measures, involving a total of nine different measures. The performance of all ensembles was then compared based on all these nine performance measures. Ensembles composed of different sets of models generated by the MOEAs were also compared.
Results: (a) Ensembles trained on LSD, MMRE and PRED (25) obtained the best results in terms of most performance measures, being considered more successful than the others. Optimising certain performance measures did not necessarily lead to the best test performance on these particular measures probably due to overfitting. (b) There was no inherent advantage in using ensembles composed of all the SEE models generated by the MOEA in comparison to using solely the best SEE model according to each measure separately.
Conclusions: Care must be taken to prevent overfitting on the performance measures being optimised. Our results suggest that concurrently optimising LSD, MMRE and PRED (25) promoted more ensemble diversity than other combinations of measures, and hence performed best. Low diversity is more likely to lead to overfitting.

References

[1]
C. Bishop. Pattern Recognition and Machine Learning. Springer, Singapore, 2006.
[2]
B. Boehm. Software Engineering Economics. Prentice-Hall, Englewood Cliffs, NJ, 1981.
[3]
B. Boehm, C. Abts, A. W. Brown, S. Chulani, B. K. Clark, E. Horowitz, R. Madachy, D. J. Reifer, and B. Steece. Software Cost Estimation with COCOMO II. Prentice-Hall, Englewood Cliffs, NJ, 2000.
[4]
G. Brown, J. Wyatt, R. Harris, and X. Yao. Diversity creation methods: A survey and categorisation. Information Fusion, 6:5--20, 2005.
[5]
A. Chandra and X. Yao. Ensemble learning using multi-objective evolutionary algorithms. Journal of Math. Modelling and Algorithms, 5(4):417--445, 2006.
[6]
H. Chen and X. Yao. Regularized negative correlation learning for neural network ensembles. TNN, 20(12):1962--1979, 2009.
[7]
K. Dejaeger, W. Verbeke, D. Martens, and B. Baesens. Data mining techniques for software effort estimation: A comparative study. TSE, 38(2):375--397, 2012.
[8]
J. Demšar. Statistical comparisons of classifiers over multiple data sets. JMLR, 7:1--30, 2006.
[9]
J. Dolado. A validation of the component-based method for software size estimation. IEEE TSE, 26:1006--1021, 2000.
[10]
J. Dolado. On the problem of the software cost function. IST, 43:61--72, 2001.
[11]
F. Ferrucci, C. Gravino, and F. Sarro. How multi-objective genetic programming is effective for software development effort estimation? In SSBSE, pages 274--275, September 2011.
[12]
T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit. A simulation study of the model evaluation criterion MMRE. TSE, 29(11):985--995, 2003.
[13]
M. Harman. The relationship between search based software engineering and predictive modeling. In PROMISE, pages 2492--2499, 2010.
[14]
M. Jorgensen and M. Shepperd. A systematic review of software development cost estimation studies. TSE, 33(1):33--53, 2007.
[15]
V. Khare, X. Yao, and K. Deb. Performance scaling of multi-objective evolutionary algorithms. In C. M. Fonseca, P. J. Fleming, E. Zitzler, K. Deb, and L. Thiele, editors, 2nd International Conference on Evolutionary Multi-Criterion Optimization (EMO'03), Lecture Notes in Computer Science, volume 2632, pages 376--390. Springer-Verlag, 2003.
[16]
E. Kocaguneli, T. Menzies, and J. Keung. On the value of ensemble effort estimation. TSE, 38:1403--1416, 2012.
[17]
Y. Kultur, B. Turhan, and A. Bener. Ensemble of neural networks with associative memory (ENNA) for estimating software development costs. Knowledge-Based Systems, 22:395--402, 2009.
[18]
L. I. Kuncheva and C. J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51:181--207, 2003.
[19]
T. Menzies, Z. Chen, J. Hihn, and K. Lum. Selecting best practices for effort estimation. TSE, 32(11):883--895, 2006.
[20]
L. Minku and X. Yao. Ensembles and locality: Insight on improving software effort estimation. IST, 55(8):1512--1528, 2013.
[21]
L. L. Minku. Machine learning for software effort estimation. The 13th CREST Open Workshop -- Future Internet Testing (FITTEST) & Search Based Software Engineering (SBSE), http://crest.cs.ucl.ac.uk/cow/13/slides/presentation_leandro.pdf, http://crest.cs.ucl.ac.uk/cow/13/videos/M2U00270Minku.mp4, May 2011.
[22]
L. L. Minku and X. Yao. Software effort estimation as a multi-objective learning problem. ACM TOSEM, 2012 (to appear), final author's version available at http://www.cs.bham.ac.uk/~minkull/publications/MinkuYaoTOSEM12.pdf.
[23]
M. Perrone and L. Cooper. Artificial Neural Networks for Speech and Vision, chapter When networks disagree: Ensemble methods for hybrid neural networks, pages 126--142. Chapman & Hall, 1993.
[24]
F. Sarro. Genetic programming for effort estimation. The 14th CREST Open Workshop -- Genetic Programming for Software Engineering), http://crest.cs.ucl.ac.uk/cow/14/slides/Sarro_ 14thCrestOW.pdf, http://crest.cs.ucl.ac.uk/cow/14/videos/M2U00285Federica.mp4, July 2011.
[25]
F. Sarro, F. Ferrucci, and C. Gravino. Single and multi objective genetic programming for software development effort estimation. In ACM SAC'12, pages 1221--1226, Riva del Garda (Trento), Italy, 2012.
[26]
A. S. Sayyad, T. Menzies, and H. Ammar. On the value of user preferences in search-based software engineering: A case study in software product lines. In ICSE, pages 492--501, 2013.
[27]
Y. Shan, R. J. McKay, C. J. Lokan, and D. L. Essam. Software project effort estimation using genetic programming. In ICCCAS & WESINO EXPO, volume 2, pages 1108--1112, Chengdu, Sichuan, 2002.
[28]
M. Shepperd and S. McDonell. Evaluating prediction systems in software project estimation. IST, 54(8):820--827, 2012.
[29]
N. Srinivas and K. Deb. Multiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary Computation, 2:221--248, 1994.
[30]
S. Wang and X. Yao. Relationships between diversity of classification ensembles and single-class performance measures. TKDE, 25(1):206--219, 2013.
[31]
Z. Wang, K. Tang, and X. Yao. Multi-objective approaches to optimal testing resource allocation in modular software systems. TR, 59(3):563--575, 2010.
[32]
J. Wen, S. Li, Z. Lin, Y. Hu, and C. Huang. Systematic literature review of machine learning based software development effort estimation models. IST, 54:41--59, 2012.
[33]
Y. Yu, X. Yao, and Z.-H. Zhou. On the approximation ability of evolutionary optimization with application to minimum set cover. AI, 180--181:20--33, 2012.

Cited By

View all
  • (2023)The Impact of Data Quality on Software Testing Effort PredictionElectronics10.3390/electronics1207165612:7(1656)Online publication date: 31-Mar-2023
  • (2023)Mitigating Unfairness via Evolutionary Multiobjective Ensemble LearningIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.320954427:4(848-862)Online publication date: 1-Aug-2023
  • (2023)Software Cost and Effort Estimation: Current Approaches and Future TrendsIEEE Access10.1109/ACCESS.2023.331271611(99268-99288)Online publication date: 2023
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences
PROMISE '13: Proceedings of the 9th International Conference on Predictive Models in Software Engineering
October 2013
103 pages
ISBN:9781450320160
DOI:10.1145/2499393
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2013

Check for updates

Author Tags

  1. ensembles of learning machines
  2. multi-objective evolutionary algorithms
  3. software effort estimation

Qualifiers

  • Research-article

Funding Sources

Conference

PROMISE '13

Acceptance Rates

Overall Acceptance Rate 98 of 213 submissions, 46%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)68
  • Downloads (Last 6 weeks)8
Reflects downloads up to 21 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2023)The Impact of Data Quality on Software Testing Effort PredictionElectronics10.3390/electronics1207165612:7(1656)Online publication date: 31-Mar-2023
  • (2023)Mitigating Unfairness via Evolutionary Multiobjective Ensemble LearningIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.320954427:4(848-862)Online publication date: 1-Aug-2023
  • (2023)Software Cost and Effort Estimation: Current Approaches and Future TrendsIEEE Access10.1109/ACCESS.2023.331271611(99268-99288)Online publication date: 2023
  • (2023)A two-stage multiobjective evolutionary ensemble learning for silicon prediction in blast furnaceComplex & Intelligent Systems10.1007/s40747-023-01199-w10:1(1639-1660)Online publication date: 19-Aug-2023
  • (2022)MEG: Multi-objective Ensemble Generation for Software Defect PredictionProceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3544902.3546255(159-170)Online publication date: 19-Sep-2022
  • (2022)Towards An Accurate Stacked Ensemble Learning Model For Thyroid Earlier Detection2022 IEEE/ACS 19th International Conference on Computer Systems and Applications (AICCSA)10.1109/AICCSA56895.2022.10017629(1-8)Online publication date: Dec-2022
  • (2022)Modeling Human Resource Experience Evolution for Multiobjective Project Scheduling in Large Scale Software ProjectsIEEE Access10.1109/ACCESS.2022.316959610(44677-44690)Online publication date: 2022
  • (2021)A Generic Analogy-Centered Software Cost Estimation Based on Differential Evolution Exploration ProcessThe Computer Journal10.1093/comjnl/bxaa19964:3(462-472)Online publication date: 1-Feb-2021
  • (2021)CSSG: A cost‐sensitive stacked generalization approach for software defect predictionSoftware Testing, Verification and Reliability10.1002/stvr.176131:5Online publication date: 8-Feb-2021
  • (2020)Better software analytics via “DUO”: Data mining algorithms using/used-by optimizersEmpirical Software Engineering10.1007/s10664-020-09808-925:3(2099-2136)Online publication date: 1-May-2020
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media