Article

Test suite prioritization by interaction coverage

Authors:

Renée C. Bryce,

Atif M. MemonAuthors Info & Claims

DOSTA '07: Workshop on Domain specific approaches to software test automation: in conjunction with the 6th ESEC/FSE joint meeting

Pages 1 - 7

https://doi.org/10.1145/1294921.1294922

Published: 04 September 2007 Publication History

Abstract

Event-driven software (EDS) is a widely used class of software that takes sequences of events as input, changes state, and outputs new event sequences. Managing the size of tests suites for EDS is difficult as the number of event combinations and sequences grow exponentially with the number of events. We propose a new testing technique that extends software interaction testing. Traditional software interaction testing systematically examines all t-way interactions of parameters for a program. This paper extends the notion to t-way interactions over sequences of events. The technique applies to many classes of software; we focus on that of EDS. As a proof-of-concept, we prioritize existing test suites for four GUI-based programs by t-way interaction coverage. We compare the rate of fault detection with that of several other prioritization criteria. Results show that prioritization by interaction coverage has the fastest rate of fault detection in half of our experiments, making the most impact when tests have high interaction coverage.

References

[1]

R. C. Bryce and C. J. Colbourn. The density algorithm for pairwise interaction testing. Journal of Software Testing, Verification, and Reliability, to appear.

Digital Library

[2]

R. C. Bryce, A. Rajan, and M. P. E. Heimdahl. Interaction testing in model-based development: Effect on model-coverage. Proc. of the 13^th Asia-Pacific Software Engineering Conf., pages 258--269, Dec. 2006.

Digital Library

[3]

C. J. Colbourn. Combinatorial aspects of covering arrays. Le Matematiche (Catania), 58:121--167, 2004.

[4]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE Trans. on Software Engineering, 18(2):159--182, 2002.

Digital Library

[5]

S. Elbaum, G. Rothermel, S. Kanduri, and A. Malishevsky. Selecting a cost-effective test case prioritization technique. Software Quality Journal, 12(3):185--210, 2004.

Digital Library

[6]

D. R. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Trans. on Software Engineering, 30(6):418--421, Oct. 2004.

Digital Library

[7]

Atif M. Memon. An event-flow model of gui-based applications for testing. Software Testing, Verification and Reliability, 2007.

Digital Library

[8]

Atif M. Memon, Mary Lou Soffa, and Martha E. Pollack. Coverage criteria for GUI testing. In ESEC/FSE-9: Proc. of the 8th European software engineering conf. held jointly with 9th ACM SIGSOFT Int. symposium on Foundations of software engineering, pages 256--267, 2001.

Digital Library

[9]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. ACM Trans. on Software Engineering and Methodology, 27(10):929--948, 2001.

Digital Library

[10]

A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In Proc. of the Int. Symposium on Software Testing and Analysis, pages 97--106, Jul. 2002.

Digital Library

[11]

Kristen R. Walcott, Mary Lou Soffa, Gregory M. Kapfhammer, and Robert S. Roos. Timeaware test suite prioritization. In Proc. of the Int. Symposium on Software Testing and Analysis, pages 1--12, Jul. 2006.

Digital Library

[12]

L. White. Regression testing of gui event interactions. In Proc. of the Int. Conf. on Software Maintenance, pages 350--358, Nov. 1996.

Digital Library

[13]

L. White and H. Almezen. Generating test cases for gui responsibilities using complete interaction sequences. In Proc. of the Interactional Symposium on Software Reliability Engineering, pages 110--121, 2000.

Digital Library

[14]

Qing Xie and Atif M. Memon. Studying the characteristics of a 'good' GUI test suite. In Proc. of the 17th IEEE Int. Symposium on Software Reliability Engineering. IEEE Computer Society Press, 2006.

Digital Library

[15]

C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. on Software Engineering, 31(1):20--34, Jan. 2006.

Digital Library

[16]

Xun Yuan and Atif M. Memon. Using GUI run-time state as feedback to generate test cases. In Proc. of the 29th Int. Conf. on Software Engineering, May 2007.

Digital Library

Cited By

Khan MMichaels RWilliams DDinal BGurkas BLuloh ABryce R(2023)Post Prioritization Techniques to Improve Code Coverage for SARSA Generated Test Cases2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC)10.1109/CCWC57344.2023.10099120(1029-1035)Online publication date: 8-Mar-2023
https://doi.org/10.1109/CCWC57344.2023.10099120
Singh ASinghrova ABhatia RRattan D(2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch7
Coppola RAlégroth E(2022)A taxonomy of metrics for GUI-based testing research: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2022.107062152(107062)Online publication date: Dec-2022
https://doi.org/10.1016/j.infsof.2022.107062
Show More Cited By

Index Terms

Test suite prioritization by interaction coverage
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Test prioritization for pairwise interaction coverage
A-MOST '05: Proceedings of the 1st international workshop on Advances in model-based testing

Interaction testing is widely used in screening for faults. In software testing, it provides a natural mechanism for testing systems to be deployed on a variety of hardware and software configurations. Several algorithms published in the literature are ...
Developing a Single Model and Test Prioritization Strategies for Event-Driven Software

Event-Driven Software (EDS) can change state based on incoming events; common examples are GUI and Web applications. These EDSs pose a challenge to testing because there are a large number of possible event sequences that users can invoke through a user ...
Aggregate-strength interaction test suite prioritization

HighlightsWe propose a new category of prioritization methods of interaction test suites.We propose a new dissimilarity measure for combinatorial test cases.Three versions of our method are evaluated using simulations and empirical studies.Our methods ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DOSTA '07: Workshop on Domain specific approaches to software test automation: in conjunction with the 6th ESEC/FSE joint meeting

September 2007

44 pages

ISBN:9781595937261

DOI:10.1145/1294921

Program Chairs:
Alan Hartman
IBM Haifa Labs
,
Mika Katara
Tampere University of Technology
,
Amit Paradkar
IBM T.J. Watson Research Center

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
CEPIS: The Council of European Professional Informatics Societies

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 September 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ESEC/FSE07

Sponsor:

ACM
SIGSOFT
CEPIS

ESEC/FSE07: Joint 11th European Software Engineering Conference 2007

September 4, 2007

Dubrovnik, Croatia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
641
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khan MMichaels RWilliams DDinal BGurkas BLuloh ABryce R(2023)Post Prioritization Techniques to Improve Code Coverage for SARSA Generated Test Cases2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC)10.1109/CCWC57344.2023.10099120(1029-1035)Online publication date: 8-Mar-2023
https://doi.org/10.1109/CCWC57344.2023.10099120
Singh ASinghrova ABhatia RRattan D(2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch7
Coppola RAlégroth E(2022)A taxonomy of metrics for GUI-based testing research: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2022.107062152(107062)Online publication date: Dec-2022
https://doi.org/10.1016/j.infsof.2022.107062
Huang RTowey DXu YZhou YYang N(2022)Dissimilarity‐based test case prioritization through data fusionSoftware: Practice and Experience10.1002/spe.306852:6(1352-1377)Online publication date: 18-Jan-2022
https://doi.org/10.1002/spe.3068
Ghannam HSayyad A(2021)Test Cases Prioritization Framework for Component Based Front End Technologies2021 Palestinian International Conference on Information and Communication Technology (PICICT)10.1109/PICICT53635.2021.00040(169-174)Online publication date: Sep-2021
https://doi.org/10.1109/PICICT53635.2021.00040
Nguyen HLe HNguyen V(2021)Prioritizing automated test cases of Web applications using reinforcement learning: an enhancement2021 13th International Conference on Knowledge and Systems Engineering (KSE)10.1109/KSE53942.2021.9648835(1-8)Online publication date: 10-Nov-2021
https://doi.org/10.1109/KSE53942.2021.9648835
Michaels RKhan MBryce R(2021)Test Suite Prioritization with Element and Event Sequences for Android Applications2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC)10.1109/CCWC51732.2021.9376143(1326-1332)Online publication date: 27-Jan-2021
https://doi.org/10.1109/CCWC51732.2021.9376143
Vos TAho PPastor Ricos FRodriguez‐Valdes OMulders A(2021) testar – scriptless testing through graphical user interface Software Testing, Verification and Reliability10.1002/stvr.177131:3Online publication date: 25-Apr-2021
https://doi.org/10.1002/stvr.1771
Chen LHassan FWang XZhang LRothermel GBae D(2020)Taming behavioral backward incompatibilities via cross-project testing and analysisProceedings of the ACM/IEEE 42nd International Conference on Software Engineering10.1145/3377811.3380436(112-124)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377811.3380436
Di Nucci DPanichella AZaidman ADe Lucia A(2020)A Test Case Prioritization Genetic Algorithm Guided by the Hypervolume IndicatorIEEE Transactions on Software Engineering10.1109/TSE.2018.286808246:6(674-696)Online publication date: 1-Jun-2020
https://doi.org/10.1109/TSE.2018.2868082
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents