Abstract
Regression testing, as an important part of the software life cycle, ensures the validity of modified software. Researchers’ focus of this research is on functional requirement-based ‘Test Case Prioritization’ (TCP) because requirement specifications help keep the software correctness on customers’ perceived priorities. This research study is aimed to investigate requirement-based TCP approaches, regression testing aspects, applications regarding the validation of proposed TCP approaches, systems’ size under regression testing, test case size and relevant revealed faults, TCP related issues, TCP issues and types of primary studies. Researchers of this paper examined research publications, which have been published between 2009 and 2019, within the seven most significant digital repositories. These repositories are popular, and mostly used for searching papers on topics in software engineering domain. We have performed a meticulous screening of research studies and selected 35 research papers through which to investigate the answers to the proposed research questions. The final outcome of this paper showed that functional requirement-based TCP approaches have been widely covered in primary studies. The results indicated that fault size and the number of test cases are mostly discussed as regression testing aspects within primary studies. In this review paper, it has been identified that iTrust system is widely examined by researchers in primary studies. This paper’s conclusion indicated that most of the primary studies have been demonstrated in the real-world settings by respective researchers of focused primary studies. The findings of this “Systematic Literature Review” (SLR) reveal some suggestions to be undertaken in future research works, such as improving the software quality, and conducting evaluations of larger systems.
Similar content being viewed by others
References
Harrold M. Reduce, reuse, recycle, recover: techniques for improved regression testing. In: SoftwareMaintenance, 2009. ICSM 2009. IEEE International Conference on 20–26 Sept. 2009, Edmonton, AB, Canada 2009.
Memon A. Advances in computers, vol. 107. Salt Lake City: Academic Press; 2017.
Hao D, Zhang L, Zang L, Wang Y, Wu X, Xie T. To be optimal or not in test-case prioritization. IEEE Trans Software Eng. 2016;42(5):490–505.
Pandey A, Goyal N. Early software reliability prediction: a fuzzy logic approach. New York: Springer; 2013.
Jiang B, Chan WK. Input-based adaptive randomized test case prioritization: a local beam search approach. J Syst Softw. 2015;105:91–106.
Parejo JA, Sánchez AB, Segura S, Ruiz-Cortés A, Lopez-Herrejon RE, Egyed A. Multi- objective test case prioritization in highly configurable systems: a case study. J Syst Softw. 2016;122:287–310.
Hao D, Zhang L, Zhang L, Rothermel G, Mei H. A unified test case prioritization approach. ACM Trans Softw Eng Methodol (TOSEM). 2014;24(2):10.
Sampath S, Bryce RE, Memon AM. A uniform representation of hybrid criteria for regression testing. IEEE Trans Softw Eng. 2013;39(10):1326–44.
Hettiarachchi C, Do H, Choi B. Risk-based test case prioritization using a fuzzy expert system. Inf Softw Technol. 2016;69:1–15.
Lin CT, Chen CD, Tsai CS, Kapfhammer GM. History-based test case prioritization with software version awareness. In: Engineering of complex computer systems (ICECCS), 2013 18th International Conference on IEEE 2013;171–2.
Gao D, Guo X, Zhao L. Test case prioritization for regression testing based on ant colony optimization. In: Software Engineering and Service Science (ICSESS), 2015. 6th IEEE International Conference on IEEE 2015; 275–9.
Khalilian A, Azgomi MA, Fazlalizadeh Y. An improved method for test case prioritization by incorporating historical test case data. Sci Comput Program. 2012;78(1):93–116.
Lewis W. Software testing and continuous quality improvement. 3rd ed. Boca Raton: CRC Press; 2016.
Agarwal B, Tayal S. Software engineering. New Delhi: Laxmi Publications; 2009.
Elbaum S, Malishevsky AG, Rothermel G. Test case prioritization: a family of empirical studies. IEEE Trans Software Eng. 2002;28(2):159–82.
Catal C, Mishra D. Test case prioritization: a systematic mapping study. Softw Qual J. 2013;21:445–78.
Qiu D, Li B, Ji S, Leung H. Regression testing of web service: a systematic mapping study. ACM Comput Surv (CSUR). 2015;47(2):21.
Khatibsyarbini M, Isa M, Jawawi DA, Tumeng R. Test case prioritization approaches in regression testing: a systematic literature review. Inf Softw Technol. 2017;93:74–93.
Kazmi R, Jawawi DN, Mohamad R, Ghani I. Effective regression test case selection: a systematic literature review. ACM Comput Surv (CSUR). 2017;50(2):29.
Rosero R, Gomez O, Rodriguez G. 15 years of software regression testing techniques—a survey. Int J Software Eng Knowl Eng. 2016;26(5):675–89.
Arora P, Bhatia R. A systematic review of agent-based test case generation for regression testing. Arab J Sci Eng. 2018;43(2):447–70.
Gallagher L, Offutt J, Cincotta A. Integration testing of object-oriented components using finite state machines. Softw Test Verif Reliab. 2006;16(4):215–66.
Mohanty S, Acharya AA, Mohapatra DP. A model based prioritization technique for component based software retesting using UML state chart diagram. In: Electronics Computer Technology (ICECT), 2011 3rd International Conference on. IEEE 2011, Vol. 2, pp. 364–8.
Kundu D, Sarma M, Samanta D, Mall R. System testing for object-oriented systems with test case prioritization. Softw Test Verif Reliab. 2009;19(4):297–333.
Zhang L, Hao D, Zhang L, Rothermel G, Mei H. Bridging the gap between the total and additional test-case prioritization strategies. In Proceedings of the 2013 International Conference on Software Engineering IEEE Press 2013; 192–201.
Salehie M, Li S, Tahvildari L, Dara R, Li S, Moore M. Prioritizing requirements-based regression test cases: A goal-driven practice. In Software Maintenance and Reengineering. (CSMR), 2011 15th European Conference on. IEEE 2011;329–32.
Arcaini P, Gargantini A, Vavassori P. Generating tests for detecting faults in feature models. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on IEEE 2015;1–10.
Bryce RC, Colbourn CJ. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf Softw Technol. 2006;48(10):960–70.
Korel B, Tahat LH, Harman M. Test prioritization using system models. In Software. Maintenance, 2005. ICSM'05. Proceedings of the 21st IEEE International Conference on. IEEE 2005;559–68.
Voinea L, Sassenburg H (2009) Improving testing efficiency: agile test case prioritization.
Jaffarur- Rehman JU, Jabeen F, Bertolino A, Polini A. Testing software components for integration: a survey of issues and techniques. Softw Test Verif Reliab. 2007;17(2):95–133.
Chen TY, Kuo FC, Liu H, Wong WE. Does adaptive random testing deliver a higher confidence than random testing?. In Quality Software, 2008. QSIC'08. The Eighth International Conference on (pp. 145–54). IEEE 2008.
Zhang L, Hou SS, Guo C, Xie T, Mei H. Time-aware test-case prioritization using integer linear programming. In Proceedings of the eighteenth international symposium on Software testing and analysis (pp. 213–24). ACM 2009.
Kumar H, Chauhan N. A module coupling slice based test case prioritization technique. Int J Modern Educ Comput Sci. 2015;7(7):8–16.
Maia CLB, Freitas FG, Souza JT. Applying search-based techniques for requirements based test case prioritization. In I WOES-CBSoft 2010; 24–31.
Iskrenovic-Momcilovic O, Micic A. Mechatronic software testing. In Telecommunications in Modern Satellite, Cable and Broadcasting Services, 2007. TELSIKS 2007. 8th International Conference on (pp. 486–9). IEEE 2007.
Rothermel G, Untch RH, Harrold MJ. Prioritizing test cases for regression testing. IEEE Trans Softw Eng. 2001;27(10):929–48.
Do H, Rothermel G. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng. 2006;32(9):733–52.
Jiang B, Zhang Z, Chan WK, Tse TH, Chen TY. How well does test case prioritization integrate with statistical fault localization? Inf Softw Technol. 2012;54:739–58.
Kitchenham B, Brereton P, David B, Mark T, Bailey J, Linkman S. Systematic literature reviews in software engineering—A systematic literature review. Inf Softw Technol. 2009;51(1):7–15.
Kitchenham BA, Brereton P, Turner M, Niazi MK, Linkman S, Pretorius R, Budgen D. Refining the systematic literature review process two participant-observer case studies. Empir Software Eng. 2010;15:618–53.
Kitchenham B. Guidelines for performing systematic literature reviews in software engineering version 2.3, Technical Report 2007.
Engstrom E, Runeson P, Skoglund M. A systematic review on regression test selection techniques. Inf Softw Technol. 2010;52(1):14–30.
Kavitha R, Kavitha VR, Kumar NS. Requirement based test case prioritization. Paper presented at the Communication Control and Computing Technologies (ICCCCT), 2010 IEEE International Conference, Ramanathapuram, India 2010.
Mary SA, Krishnamoorthi R. Time-aware and weighted fault severity based metrics for test case prioritization. Int J Software Eng Knowl Eng. 2011;21(1):129–42.
Srikanth H, Banerjee S. Improving test efficiency through system test prioritization. J Syst Softw. 2012;85:1176–87.
Islam M, Marchetto A, Susi A, Scanniello G. A multi-objective technique to prioritize test cases based on latent semantic indexing. In 2012 16th European Conference on Software Maintenance and Reengineering. 2012;21–30.
Arafeen MJ, Do H. Test Case Prioritization Using Requirements-Based Clustering. Paper presented at the IEEE Sixth International Conference on Software Testing, Verification and Validation, Luxembourg, Luxembourg 2013.
Srikanth H, Hettiarachchi C, Do H. Requirements based test prioritization using risk factors: an industrial study. Inf Softw Technol. 2016;69:71–83.
Ma T, Zeng H, Wang X. Test case prioritization based on requirement correlations. Paper presented at the Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 2016 17th IEEE/ACIS International Conference, Shanghai, China 2016.
Wang X, Zeng H. History-based dynamic test case prioritization for requirement properties in regression testing. Paper presented at the Continuous Software Evolution and Delivery (CSED), IEEE/ACM International Workshop, Austin, TX, USA 2016.
Marchetto A, Islam MM, Asghar W, Susi A, Scanniello G. A multi-objective technique to prioritize test cases. IEEE Trans Softw Eng. 2016;42(10):918–40.
Yadav DK, Dutta S. Test case prioritization technique based on early fault detection using fuzzy logic Sign In orPurchase. Paper presented at the Computing for Sustainable Global Development (INDIACom), 2016 3rd International Conference. New Delhi, India 2016.
Haraty RA, Mansour N, Moukaha L, Khalil I. Regression test cases prioritization using clustering and code change relevance. Int J Softw Eng Knowl Eng. 2016;26(5):733–68.
Kandil P, Moussa S, Badr N. Cluster-based test cases prioritization and selection technique for agile regression testing. J Softw Evol Process. 2017;29:1–19.
Hasan MA, Rahman MA, Siddik S. Test case prioritization based on dissimilarity clustering using historical data analysis. Paper presented at the International Conference on Information, Communication and Computing Technology, Singapore 2017.
Tumeng R, Jawawi DNA, Isa MA. Test case prioritization with textual comparison metrics. In Software Engineering Conference (MySEC), 2015 9th Malaysian (pp. 7–12). IEEE 2015.
Krishnamoorthi R, Mary SSA. Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf Softw Technol. 2009;51(4):799–808.
Bjarnason E, Unterkalmsteiner M, Borg M, Engström E. A multi-case study of agile requirements engineering and the use of test cases as requirements. Inf Softw Technol. 2016;77:61–79.
Garousi V, Ozkan R, Betin-Can A. Multi-objective regression test selection in practice: an empirical study in the defense industry. Inf Softw Technol. 2018;103:40–54.
Tsai WT, Bai X, Paul R, Yu L. Scenario-based functional regression testing. In Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International (pp. 496–501). IEEE 2001.
Salem YI, Hassan R. Requirement-based test case generation and prioritization. In Computer Engineering Conference (ICENCO), 2010 International (pp. 152–7). IEEE 2010.
Tahvili S, Ahlberg M, Fornander E, Afzal W, Saadatmand M, Bohlin M, Sarabi M. Functional Dependency Detection for Integration Test Cases. In Companion of the 18th IEEE International Conference on Software Quality, Reliability, and Security 2018.
Vescan A, Şerban C, Chisăliţă-Cretu C, Dioşan L. Requirement dependencies-based formal approach for test case prioritization in regression testing. In: Intelligent Computer Communication and Processing (ICCP), 2017 13th IEEE International Conference on. IEEE 2017:181–8.
Ahmad SF, Singh DK, Suman P. Prioritization for regression testing using ant colony optimization based on test factors. In: Intelligent Communication, Control and Devices. Springer, Singapore; 2018:1353–60.
Kumar S, Ranjan P, Rajesh R. A concept for test case prioritization based upon the priority information of early phase. In: Proceedings of the International Conference on Signal, Networks, Computing, and Systems. Springer, New Delhi, 2016;213–23.
Mahali P, Mohapatra DP. Model based test case prioritization using UML behavioural diagrams and association rule mining. Int J Sys Assur Eng Manag. 2018;2:1–17.
Gupta V, Chauhan DS, Dutta K. Regression testing-based requirement prioritization of mobile applications. Int J Syst Serv Orient Eng (IJSSOE). 2012;3(4):20–39.
Liu W, Wu X, Zhang W, Xu Y. The research of the test case prioritization algorithm for black box testing. Paper presented at the Software Engineering and Service Science (ICSESS), 2014 5th IEEE International Conference Beijing, China, 2014.
Kumar S, Ranjan P, Rajesh R. An overview of test case optimization using Meta-heuristic approach. In: Recent advances in mathematics, statistics and computer science. 2016:475–85.
Pandey AK, Goyal NK. Reliability centric test case prioritization. In: Early software reliability prediction. India: Springer; 2013. p. 105–15.
Hillah LM, Maesano A-P, Rosa FD, Kordon F, Wuillemin P-H, Fontanelli R, Maesano L. Automation and intelligent scheduling of distributed system functional testing. Int J Softw Tools Technol Transfer. 2017;19:281–308.
Chen Y, Probert RL, Sims DP. Specification-based regression test selection with risk analysis. In: Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research (p. 1) (2002). IBM Press.
Do H, Hossain M. An efficient regression testing approach for PHP web applications: a controlled experiment. Softw Test Verif Reliab. 2014;24:367–85.
Barr E, Harman M, McMinn P, Shahbaz M, Yoo S. The oracle problem in software testing: a survey. IEEE Trans Softw Eng. 2015;41(5):507–25.
Yoo S, Harman M. Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab. 2012;22:67–120.
Laranjeiro N, Vieira M, Madeira H. A robustness testing approach for SOAPWeb services. J Internet Serv Appl. 2012;3:215–32.
Srikanth H, Williams L, Osborne J. System test case prioritization of new and regression test cases. In: Empirical Software Engineering, 2005. 2005 International Symposium on (pp. 10-pp). (2005) IEEE.
Zhou ZQ, Zhang S, Hagenbuchner M, Tse TH, Kuo FC, Chen TY. Automated functional testing of online search services. Softw Test Verif Reliab. 2012;22(4):221–43.
Wohlin C, Aurum A. Towards a decision-making structure for selecting a research design in empirical software engineering. Empir Software Eng. 2015;20:1427–55.
Alves V, Niu N, Alves C, Valenca G. Requirements engineering for software product lines: a systematic literature review. Inf Softw Technol. 2010;52:806–20.
Gotlieb A, Marijan D. Using global constraints to automate regression testing. AI Mag. 2017;38(1):73.
Jiang B, Tse TH, Grieskamp W, Kicillof N, Cao Y, Li X, Chan WK. Assuring the model evolution of protocol software specifications by regression testing process improvement. Softw Pract Exp. 2011;41(10):1073–103.
Piazolo F, Felderer M. Innovation and future of enterprise information systems. New York: Springer; 2013.
Mukherjee R, Patnaik KS. A survey on different approaches for software test case prioritization. J King Saud Univ Comput Inf Sci. 2018;2:2.
de Souza Neto JB, Moreira AM, Musicante MA. Semantic web services testing: a systematic mapping study. Comput Sci Rev. 2018;28:140–56.
Arrieta A, Wang S, Sagardui G, Etxeberria L. Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. J Syst Softw. 2019;149:1–34.
Do H, Mirarab S, Tahvildari L, Rothermel G. The effects of time constraints on test case prioritization: a series of controlled experiments. IEEE Trans Softw Eng. 2010;36(5):593–617.
Singh Y, Kaur A, Suri B, Singhal S. Systematic literature review on regression test prioritization techniques. Informatica. 2012;36:4.
Andrews A, Alhaddad A, Boukhris S. Black-Box model-based regression testing of fail-Safe behavior in web applications. J Syst Softw. 2019;149:318–39.
Wang S, Huang K. Improving the efficiency of functional verification based on test prioritization. Microprocess Microsyst. 2016;41:1–11.
Zhu Q, Panichella A, Zaidman A. A systematic literature review of how mutation testing supports quality assurance processes. Softw Test Verif Reliab. 2018;28(6):e1675.
Ayav T. Prioritizing MCDC test cases by spectral analysis of Boolean functions. Softw Test Verif Reliab. 2017;27(7):e1641.
Awedikian R, Yannou B. A practical model-based statistical approach for generating functional test cases: application in the automotive industry. Softw Testing Verif Reliab. 2014;24(2):85–123.
Hemmati H, Fang Z, Mäntylä MV, Adams B. Prioritizing manual test cases in rapid release environments. Softw Test Verif Reliab. 2017;27(6):e1609.
Lity S, Nieke M, Thüm T, Schaefer I. Retest test selection for product-line regression testing of variants and versions of variants. J Syst Softw. 2019;147:46–63.
Ansari A, Khan A, Khan A, Mukadam K. Optimized regression test using test case prioritization. Proc Comput Sci. 2016;79:152–60.
Wang X, Zeng H, Gao H, Miao H, Lin W. Location-based test case prioritization for software embedded in mobile devices using the law of gravitation. Mobile Inf Syst. 2019;2019(2):2.
Nooraei Abadeh M, Mirian-Hosseinabadi SH. Delta-based regression testing: a formal framework towards model-driven regression testing. J Softw Evol Process. 2015;27(12):913–52.
Mardani A, Van Fan Y, Nilashi M, Hooker RE, Ozkul S, Streimikiene D, Loganathan N. A two-stage methodology based on ensemble adaptive neuro-fuzzy inference system to predict carbon dioxide emissions. J Clean Prod. 2019;231:446–61.
Neto PADMS, Machado I, McGregor JD, De Almeida ES, de Lemos Meira SR. A systematic mapping study of software product lines testing. Inf Softw Technol. 2011;53(5):407–23.
Ahmad T, Iqbal J, Ashraf A, Truscan D, Porres I. Model-based testing using UML activity diagrams: a systematic mapping study. Comput Sci Rev. 2019;33:98–112.
Banias O. Test case selection-prioritization approach based on memoization dynamic programming algorithm. Inf Softw Technol. 2019;2:2.
Yoo S, Harman M, Tonella P, Susi A. Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proceedings of the eighteenth international symposium on Software testing and analysis (pp. 201–12). 2009. ACM.
Ledru Y, Petrenko A, Boroday S, Mandran N. Prioritizing test cases with string distances. Autom Softw Eng. 2012;19(1):65–95.
Harikarthik S, Palanisamy V, Ramanathan P. Optimal test suite selection in regression testing with testcase prioritization using modified Ann and Whale optimization algorithm. Clust Comput. 2019;22(5):11425–34.
Gupta N, Yadav V, Singh M. Automated regression test case generation for web application: a survey. ACM Comput Surv (CSUR). 2018;51(4):1–25.
Kim J, Jeong H, Lee E. Failure history data-based test case prioritization for effective regression test. in Proceedings of the Symposium on Applied Computing. 2017.
Shin SY, et al. Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach. in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2018.
Dahiya O, Solanki K. Prevailing standards in requirement-based test case prioritization: an overview. ICT Anal Appl. 2020;2:467–74.
Jahan H, Feng Z, Mahmud S. Risk-based test case prioritization by correlating system methods and their associated risks. Arabi J Sci Eng. 2020;2:2.
Paygude P, Joshi SD. Use of evolutionary algorithm in regression test case prioritization: a review. In: International conference on computer networks, big data and IoT. Springer, 2018.
Bajaj A, Sangwan OP. A systematic literature review of test case prioritization using genetic algorithms. IEEE Access. 2019;7:126355–75.
Parsons D, Susnjak T, Lange M. Influences on regression testing strategies in agile software development environments. Software Qual J. 2014;22(4):717–39.
Sun X, et al. ComboRT: a new approach for generating regression test cases for evolving programs. Int J Software Eng Knowl Eng. 2016;26(06):1001–26.
Bian Y, et al. Concrete hyperheuristic framework for test case prioritization. J Softw Evol Process. 2018;30(11):e1992.
Ahmed BS, Abdulsamad TS, Potrus MY. Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the cuckoo search algorithm. Inf Softw Technol. 2015;66:13–29.
Miranda B, et al. Fast approaches to scalable similarity-based test case prioritization. in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 2018. IEEE.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hasnain, M., Pasha, M.F., Ghani, I. et al. Functional Requirement-Based Test Case Prioritization in Regression Testing: A Systematic Literature Review. SN COMPUT. SCI. 2, 421 (2021). https://doi.org/10.1007/s42979-021-00821-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42979-021-00821-3