Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Predicting Delivery Capability in Iterative Software Development

Published: 01 June 2018 Publication History

Abstract

Iterative software development has become widely practiced in industry. Since modern software projects require fast, incremental delivery for every iteration of software development, it is essential to monitor the execution of an iteration, and foresee a capability to deliver quality products as the iteration progresses. This paper presents a novel, data-driven approach to providing automated support for project managers and other decision makers in predicting delivery capability for an ongoing iteration. Our approach leverages a history of project iterations and associated issues, and in particular, we extract characteristics of previous iterations and their issues in the form of features. In addition, our approach characterizes an iteration using a novel combination of techniques including feature aggregation statistics, automatic feature learning using the Bag-of-Words approach, and graph-based complexity measures. An extensive evaluation of the technique on five large open source projects demonstrates that our predictive models outperform three common baseline methods in Normalized Mean Absolute Error and are highly accurate in predicting the outcome of an ongoing iteration.

References

[1]
B. Michael, S. Blumberg, and J. Laartz, “Delivering large-scale IT projects on time, on budget, and on value,” McKinsey Quarterly, 2012 . [Online]. Available: http://www.mckinsey.com/business-functions/digital-mckinsey/ ourinsights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value, Accessed: Mar. 2014.
[2]
B. Flyvbjerg and A. Budzier, “Why your IT project may be riskier than you think,” Harvard Business Rev., vol. 89, no. 9, pp. 601–603, 2011.
[3]
L. Williams, “What agile teams think of agile principles,” Commun. ACM, vol. 55, no. 4, 2012, Art. no.
[4]
M. Cohn, Agile Estimating and Planning. Upper Saddle River, NJ, USA: Pearson Education, 2005.
[5]
A. Mockus, D. Weiss, and P. Z. P. Zhang, “ Understanding and predicting effort in software projects,” in Proc. 25th Int. Conf. Softw. Eng., 2003, pp. 274–284.
[6]
F. Sarro, A. Petrozziello, and M. Harman, “ Multi-objective software effort estimation,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 619–630.
[7]
E. Kocaguneli, T. Menzies, and J. W. Keung, “On the value of ensemble effort estimation,” IEEE Trans. Softw. Eng. , vol. 38, no. 6, pp. 1403–1416, Nov./Dec. 2012.
[8]
L. Huang, D. Port, L. Wang, T. Xie, and T. Menzies, “Text mining in supporting software systems risk assurance,” in Proc. IEEE/ACM Int. Conf. Autom. Softw. Eng., 2010, pp. 163–167.
[9]
Z. Xu, B. Yang, and P. Guo, “Software risk prediction based on the hybrid algorithm of genetic algorithm and decision tree,” Adv. Intell. Comput. Theories Appl. Aspects Contemporary Intell. Comput. Techn., vol. 2, pp. 266 –274, 2007.
[10]
Y. Hu, X. Zhang, E. Ngai, R. Cai, and M. Liu, “Software project risk analysis using Bayesian networks with causality constraints,” Decision Support Syst., vol. 56, pp. 439–449, 2013.
[11]
C. Fang and F. Marle, “A simulation-based risk network model for decision support in project risk management,” Decision Support Syst., vol. 52, no. 3, pp. 635–644, 2012.
[12]
M. N. MorenoGarcía, I. R. Román, F. J. García Peñalvo, and M. T. Bonilla, “ An association rule mining method for estimating the impact of project management policies on software quality, development time and effort,” Expert Syst. Appl., vol. 34, no. 1, pp. 522–529, 2008.
[13]
H. F. Cervone, “Understanding agile project management methods using Scrum,” OCLC Syst. Services: Int. Dig. Library Perspectives, vol. 27, no. 1, pp. 18–22, 2011.
[14]
M. Choetkiertikul, H. K. Dam, T. Tran, and A. Ghose, “Characterization and prediction of issue-related risks in software projects,” in Proc. 12th Working Conf. Mining Softw. Repositories, 2015, pp. 280–291.
[15]
S. Kaufman and C. Perlich, “Leakage in data mining: Formulation, detection, and avoidance,” ACM Trans. Knowl. Discovery Data, vol. 6, no. 15, pp. 556 –563, 2012.
[16]
L. D. Panjer, “Predicting eclipse bug lifetimes,” in Proc. 4th Int. Workshop Mining Softw. Repositories, 2007, pp. 29 –32.
[17]
X. Xia, D. Lo, E. Shihab, X. Wang, and X. Yang, “ELBlocker: Predicting blocking bugs with ensemble imbalance learning,” Inf. Softw. Technol., vol. 61, pp. 93– 106, 2015.
[18]
M. Choetkiertikul, H. K. Dam, T. Tran, and A. Ghose, “Predicting delays in software projects using networked classification,” in Proc. 30th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2015, pp. 353–364.
[19]
H. Valdivia Garcia, E. Shihab, and H. V. Garcia, “ Characterizing and predicting blocking bugs in open source projects,” in Proc. 11th Working Conf. Mining Softw. Repositories, 2014, pp. 72–81.
[20]
X. Xia, D. Lo, M. Wen, E. Shihab, and B. Zhou, “An empirical study of bug report field reassignment,” in Proc. Conf. Softw. Maintenance Reengineering Reverse Eng., 2014, pp. 174 –183.
[21]
D. R. McCallum and J. L. Peterson, “Computer-based readability indexes,” in Proc. ACM Conf., 1982, pp. 44–48.
[22]
P. Hooimeijer and W. Weimer, “Modeling bug report quality,” in Proc. 22nd IEEE/ACM Int. Conf. Autom. Softw. Eng., Nov. 2007, pp. 34– 44.
[23]
P. Tirilly, V. Claveau, and P. Gros, “Language modeling for bag-of-visual words image categorization,” in Proc. Int. Conf. Content-Based Image Video Retrieval, 2008, pp. 249–258.
[24]
E. Zegura, K. Calvert, and S. Bhattacharjee, “How to model an internetwork,” in Proc. 15th Annu. Joint Conf. IEEE Comput. Society's Netw. Next Generation, 1996, pp. 594– 602.
[25]
P. Bhattacharya, M. Iliofotou, I. Neamtiu, and M. Faloutsos, “Graph-based analysis and prediction for software evolution,” in Proc. 34th Int. Conf. Softw. Eng., 2012, pp. 419–429.
[26]
L. Rokach, “Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography,” Comput. Statistics Data Anal., vol. 53, no. 12, pp. 4046–4072, 2009.
[27]
D. O. Maclin and R., “ Popular ensemble methods: An empirical study,” J. Artif. Intell. Res., vol. 11, pp. 169–198, 1999.
[28]
T. Dietterich, “Ensemble methods in machine learning,” in Proc. 1st Int. Workshop Multiple Classifier Syst., 2000, pp. 1 –15.
[29]
T. G. Dietterich, “An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization,” Mach. Learning , vol. 40, no. 2, pp. 139–157, 2000.
[30]
M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera, “A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches,” IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev., vol. 42, no. 4, pp. 463– 484, 2012.
[31]
S. M. Halawani, I. A. Albidewi, and A. Ahmad, “ A novel ensemble method for regression via classification problems,” J. Comput. Sci. , vol. 7, no. 3, pp. 387–393, 2011.
[32]
L. Breiman, “Random forests,” Mach. Learning , vol. 45, no. 1, pp. 5–32, 2001.
[33]
J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” Ann. Statistics, vol. 29, no. 5, pp. 1189–1232, 2001.
[34]
J. H. Friedsman, “Stochastic gradient boosting,” Comput. Statistics Data Anal., vol. 38, no. 4, pp. 367– 378, 2002.
[35]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” J. Mach. Learning Res., vol. 15, pp. 1929 –1958, 2014.
[36]
X. Wang and A. McCallum, “Topics over time: A non-Markov continuous-time model of topical trends,” in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2006, pp. 424–433.
[37]
M. Fernández-Delgado, E. Cernadas, S. Barro, D. Amorim, and D. Amorim Fernández-Delgado, “Do we need hundreds of classifiers to solve real world classification problems?” J. Mach. Learning Res., vol. 15, pp. 3133–3181, 2014.
[38]
K. H. Esbensen and P. Geladi, “Principles of proper validation: Use and abuse of re-sampling for validation,” J. Chemometrics, vol. 24, pp. 168–187, 2010.
[39]
K. Goldberg, T. Roeder, D. Gupta, and C. Perkins, “Eigentaste: A constant time collaborative filtering algorithm,” Inf. Retrieval, vol. 4, no. 2, pp. 133–151, 2001.
[40]
K. Muller, “Statistical power analysis for the behavioral sciences,” Technometrics, vol. 31, no. 4, pp. 499 –500, 1989.
[41]
A. Arcuri and L. Briand, “A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering,” Softw. Testing Verification Rel., vol. 24, no. 3, pp. 219–250, 2014.
[42]
S. Baccianella, A. Esuli, and F. Sebastiani, “ Evaluation measures for ordinal regression,” in Proc. 9th Int. Conf. Intell. Syst. Des. Appl., 2009, pp. 283–287.
[43]
D. M. W. Powers, “Evaluation: From precision, recall and F-factor to ROC, informedness, markedness & correlation david,” J. Mach. Learning Technol., vol. 2, no. 1, pp. 37–63, 2011.
[44]
R. Genuer, J.-M. Poggi, and C. Tuleau-Malot, “ Variable selection using random forests,” Pattern Recognit. Lett., vol. 31, no. 14, pp. 2225–2236, 2010.
[45]
K. Tumer and N. C. Oza, “Input decimated ensembles,” Pattern Anal. Appl., vol. 6, no. 1, pp. 65–77, 2003.
[46]
K. Tumer and J. Ghosh, “Error correlation and error reduction in ensemble classifiers,” Connection Sci., vol. 8, no. 3–4, pp. 385 –404, 1996.
[47]
E. Tuv, “Feature selection with ensembles, artificial variables, and redundancy elimination,” J. Mach. Learning Res., vol. 10, pp. 1341 –1366, 2009.
[48]
M. Shepperd and S. MacDonell, “Evaluating prediction systems in software project estimation,” Inf. Softw. Technol., vol. 54, no. 8, pp. 820–827, 2012. [Online]. Available: https://doi.org/10.1016/j.infsof.2011.12.008
[49]
S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking classification models for software defect prediction: A proposed framework and novel findings,” IEEE Trans. Softw. Eng., vol. 34, no. 4, pp. 485–496, Jul. 2008.
[50]
M. Usman, E. Mendes, F. Weidt, and R. Britto, “Effort estimation in agile software development: A systematic literature review,” in Proc. 10th Int. Conf. Predictive Models Softw. Eng., 2014, pp. 82–91.
[51]
M. Usman, E. Mendes, and J. Börstler, “ Effort estimation in agile software development: A survey on the state of the practice,” in Proc. 19th Int. Conf. Eval. Assessment Softw. Eng., 2015, pp. 1 –10.
[52]
P. Abrahamsson, R. Moser, W. Pedrycz, A. Sillitti, and G. Succi, “Effort prediction in iterative software development processes – incremental versus global prediction models,” in Proc. 1st Int. Symp. Empirical Softw. Eng. Measurement, 2007, pp. 344–353.
[53]
B. W. Boehm, R. Madachy, and B. Steece, Software Cost Estimation with Cocomo II. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2000.
[54]
O. Benediktsson, D. Dalcher, K. Reed, and M. Woodman, “COCOMO-based effort estimation for iterative and incremental software development,” Softw. Quality J., vol. 11, pp. 265–281, 2003.
[55]
P. Hearty, N. Fenton, D. Marquez, and M. Neil, “Predicting project velocity in XP using a learning dynamic Bayesian network model,” IEEE Trans. Softw. Eng. , vol. 35, no. 1, pp. 124–137, Jan./Feb. 2009.
[56]
R. Torkar, N. M. Awan, A. K. Alvi, and W. Afzal, “Predicting software test effort in iterative development using a dynamic Bayesian network,” in Proc. 21st IEEE Int. Symp. Softw. Rel. Eng, Industry Practice Track, 2010, pp. 1 –10.
[57]
M. Perkusich, H. De Almeida, and A. Perkusich, “ A model to detect problems on scrum-based software development projects,” in Proc. ACM Symp. Appl. Comput., 2013, pp. 1037–1042.
[58]
P. Bhattacharya and I. Neamtiu, “Bug-fix time prediction models: can we do better?” in Proc. 8th Working Conf. Mining Softw. Repositories, 2011, pp. 207– 210.
[59]
E. Giger, M. Pinzger, and H. Gall, “Predicting the fix time of bugs,” in Proc. 2nd Int. Workshop Recommendation Syst. Softw. Eng. , 2010, pp. 52–56.
[60]
L. Marks, Y. Zou, and A. E. Hassan, “Studying the fix-time for bugs in large open source projects,” in Proc. 7th Int. Conf. Predictive Models Softw. Eng., 2011, pp. 1–8.
[61]
C. Weiss, R. Premraj, T. Zimmermann, and A. Zeller, “How long will it take to fix this bug?” in Proc. 4th Int. Workshop Mining Softw. Repositories, 2007, pp. 1–8.
[62]
N. Bettenburg and A. E. Hassan, “Studying the impact of dependency network measures on software quality,” in Proc. Int. Conf. Softw. Maintenance, 2012, pp. 1– 10.
[63]
T. Zimmermann and N. Nagappan, “Predicting defects using network analysis on dependency graphs,” in Proc. 13th Int. Conf. Softw. Eng., 2008, pp. 531– 540.
[64]
T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy, “Characterizing and predicting which bugs get reopened,” in Proc. 34th Int. Conf. Softw. Eng., Jun. 2012, pp. 1074–1083.
[65]
E. Shihab, et al., “Studying re-opened bugs in open source software,” Empirical Softw. Eng., vol. 18, no. 5, pp. 1005–1042, Sep. 2012.
[66]
A. Lamkanfi, S. Demeyer, E. Giger, and B. Goethals, “Predicting the severity of a reported bug,” in Proc. 7th IEEE Working Conf. Mining Softw. Repositories , 2010, pp. 1–10.
[67]
T. Menzies and A. Marcus, “Automated severity assessment of software defect reports,” in Proc. Int. Conf. Softw. Maintenance, 2008, pp. 346–355.
[68]
D. Alencar, et al., “An empirical study of delays in the integration of addressed issues,” in Proc. Int. Conf. Softw. Maintenance Evol., 2014, pp. 281–290.
[69]
J. Anvik and G. C. Murphy, “Reducing the effort of bug report triage,” ACM Trans. Softw. Eng. Methodology, vol. 20, no. 3, pp. 1–35, 2011.
[70]
M. M. Rahman, G. Ruhe, and T. Zimmermann, “ Optimized assignment of developers for fixing bugs an initial evaluation for eclipse projects,” in Proc. 3rd Int. Symp. Empirical Softw. Eng. Measurement, Oct. 2009, pp. 439–442.
[71]
G. Murphy and D. Čubranić, “Automatic bug triage using text categorization,” in Proc. 16th Int. Conf. Softw. Eng. Knowl. Eng., 2004, pp. 92 –97.
[72]
P. Runeson, M. Alexandersson, and O. Nyholm, “ Detection of duplicate defect reports using natural language processing,” in Proc. 29th Int. Conf. Softw. Eng., 2007, pp. 499– 510.
[73]
X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun, “An approach to detecting duplicate bug reports using natural language and execution information,” in Proc. 30th Int. Conf. Softw. Eng., 2008, pp. 461–470.
[74]
N. Bettenburg, R. Premraj, and T. Zimmermann, “ Duplicate bug reports considered harmful ... really?” in Proc. Int. Conf. Softw. Maintenance, 2008, pp. 337–345.
[75]
C. Sun, D. Lo, S. C. Khoo, and J. Jiang, “Towards more accurate retrieval of duplicate bug reports,” in Proc. 26th IEEE/ACM Int. Conf. Autom. Softw. Eng, 2011, pp. 253–262.
[76]
N. Jalbert and W. Weimer, “Automated duplicate detection for bug tracking systems,” in Proc. Int. Conf. Dependable Syst. Netw. FTCS DCC (DSN), 2008, pp. 52– 61.
[77]
A. T. Nguyen, T. T. T. N. Nguyen, D. Lo, and C. Sun, “Duplicate bug report detection with a combination of information retrieval and topic modeling,” in Proc. 27th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2012, pp. 70–79.
[78]
F. Palomba, G. Bavota, M. Di Penta, R. Oliveto, A. De Lucia, and D. Poshyvanyk, “Detecting bad smells in source code using change history information,” in Proc. 28th IEEE/ACM Int. Conf. Autom. Softw. Eng, 2013, pp. 268–278.
[79]
T. Jiang, L. Tan, and S. Kim, “Personalized defect prediction,” in Proc. 28th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2013, pp. 279–289.
[80]
S. D. Conte, H. E. Dunsmore, and V. Y. Shen, Softw. Eng. Metrics Models. Redwood City, CA, USA: Benjamin-Cummings Publishing Co., Inc., 1986.
[81]
T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit, “A simulation study of the model evaluation criterion MMRE,” IEEE Trans. Softw. Eng. , vol. 29, no. 11, pp. 985–995, Nov. 2003.
[82]
B. Kitchenham, L. Pickard, S. MacDonell, and M. Shepperd, “What accuracy statistics really measure,” IEEE Proc. Softw., vol. 148, no. 3, p. 81, 2001.
[83]
M. Korte and D. Port, “Confidence in software cost estimation results based on MMRE and PRED,” in Proc. 4th Int. Workshop Predictor Models Softw. Eng., 2008, pp. 63 –70.
[84]
D. Port and M. Korte, “Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research,” in Proc. 2nd ACM-IEEE Int. Symp. Empirical Softw. Eng. Measurement, 2008, pp. 51–60.
[85]
A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proc. 33rd Int. Conf. Softw. Eng., 2011, pp. 1–10.
[86]
R. M. Everson and J. E. Fieldsend, “Multi-class ROC analysis from a multi-objective optimisation perspective,” Pattern Recognit. Lett., vol. 27, no. 8, pp. 918–927, 2006.
[87]
B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme,” Biochimica et Biophysica Acta (BBA)-Protein Structure, vol. 405, no. 2, pp. 442–451, 1975.
[88]
G. Jurman, S. Riccadonna, and C. Furlanello, “ A comparison of MCC and CEN error measures in multi-class prediction,” PLoS One, vol. 7, no. 8, pp. 1–8, 2012.

Cited By

View all
  • (2024)Forecasting software indicators: an industry-academia collaborationEmpirical Software Engineering10.1007/s10664-024-10508-x29:6Online publication date: 23-Sep-2024
  • (2023)Dynamic Prediction of Delays in Software Projects using Delay Patterns and Bayesian ModelingProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616328(1012-1023)Online publication date: 30-Nov-2023
  • (2023)Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication StudyIEEE Transactions on Software Engineering10.1109/TSE.2022.322873949:4(2677-2697)Online publication date: 1-Apr-2023
  • Show More Cited By

Index Terms

  1. Predicting Delivery Capability in Iterative Software Development
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image IEEE Transactions on Software Engineering
    IEEE Transactions on Software Engineering  Volume 44, Issue 6
    June 2018
    101 pages

    Publisher

    IEEE Press

    Publication History

    Published: 01 June 2018

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 21 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Forecasting software indicators: an industry-academia collaborationEmpirical Software Engineering10.1007/s10664-024-10508-x29:6Online publication date: 23-Sep-2024
    • (2023)Dynamic Prediction of Delays in Software Projects using Delay Patterns and Bayesian ModelingProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616328(1012-1023)Online publication date: 30-Nov-2023
    • (2023)Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication StudyIEEE Transactions on Software Engineering10.1109/TSE.2022.322873949:4(2677-2697)Online publication date: 1-Apr-2023
    • (2022)Towards reliable agile iterative planning via predicting documentation changes of work itemsProceedings of the 19th International Conference on Mining Software Repositories10.1145/3524842.3528445(35-47)Online publication date: 23-May-2022
    • (2022)A versatile dataset of agile open source software projectsProceedings of the 19th International Conference on Mining Software Repositories10.1145/3524842.3528029(707-711)Online publication date: 23-May-2022
    • (2022)A Systematic Literature Review on the Use of Deep Learning in Software Engineering ResearchACM Transactions on Software Engineering and Methodology10.1145/348527531:2(1-58)Online publication date: 4-Mar-2022
    • (2022)Story points changes in agile iterative developmentEmpirical Software Engineering10.1007/s10664-022-10192-927:6Online publication date: 10-Aug-2022
    • (2021)Investigating documented information for accurate effort estimation in agile software developmentProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473106(1605-1609)Online publication date: 20-Aug-2021
    • (2021)Modeling team dynamics for the characterization and prediction of delays in user storiesProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678939(991-1002)Online publication date: 15-Nov-2021

    View Options

    View options

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media