research-article

Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software

Authors:

Qing XieAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 31, Issue 10

Pages 884 - 896

https://doi.org/10.1109/TSE.2005.117

Published: 01 October 2005 Publication History

Abstract

Software is increasingly being developed/maintained by multiple, often geographically distributed developers working concurrently. Consequently, rapid-feedback-based quality assurance mechanisms such as daily builds and smoke regression tests, which help to detect and eliminate defects early during software development and maintenance, have become important. This paper addresses a major weakness of current smoke regression testing techniques, i.e., their inability to automatically (re)test graphical user interfaces (GUIs). Several contributions are made to the area of GUI smoke testing. First, the requirements for GUI smoke testing are identified and a GUI smoke test is formally defined as a specialized sequence of events. Second, a GUI smoke regression testing process called Daily Automated Regression Tester (DART) that automates GUI smoke testing is presented. Third, the interplay between several characteristics of GUI smoke test suites including their size, fault detection ability, and test oracles is empirically studied. The results show that: 1) the entire smoke testing process is feasible in terms of execution time, storage space, and manual effort, 2) smoke tests cannot cover certain parts of the application code, 3) having comprehensive test oracles may make up for not having long smoke test cases, and 4) using certain oracles can make up for not having large smoke test suites.

References

[1]

A.M. Memon and Q. Xie, “Empirical Evaluation of the Fault-Detection Effectiveness of Smoke Regression Test Cases for GUI-Based Software,” Proc. Int'l Conf. Software Maintenance 2004 (ICSM '04), pp. 8-17, Sept. 2004.

Digital Library

[2]

A. Memon, A. Nagarajan, and Q. Xie, “Automating Regression Testing for Evolving GUI Software,” J. Software Maintenance and Evolution: Research and Practice, vol. 17, no. 1, pp. 27-64, 2005.

Digital Library

[3]

A.M. Memon, I. Banerjee, and A. Nagarajan, “GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing,” Proc. 10th Working Conf. Reverse Eng., pp. 260-269, Nov. 2003.

Digital Library

[4]

A.M. Memon, M.E. Pollack, and M.L. Soffa, “Hierarchical GUI Test Case Generation Using Automated Planning,” IEEE Trans. Software Eng., vol. 27, no. 2, pp. 144-155, Feb. 2001.

Digital Library

[5]

A.M. Memon, I. Banerjee, and A. Nagarajan, “What Test Oracle Should I Use for Effective GUI Testing?,” Proc. IEEE Int'l Conf. Automated Software Eng., pp. 164-173, Oct. 2003.

Digital Library

[6]

E.-A. Karlsson, L.-G. Andersson, and P. Leion, “Daily Build and Feature Development in Large Distributed Projects,” Proc. 22nd Int'l Conf. Software Eng., pp. 649-658, 2000.

Digital Library

[7]

S. McConnell, “Best Practices: Daily Build and Smoke Test,” IEEE Software, vol. 13, no. 4, pp. 143-144, July 1996.

Digital Library

[8]

K. Olsson, “Daily Build— The Best of Both Worlds: Rapid Development and Control,” technical report, Swedish Eng. Industries, 1999.

[9]

J. Robbins, Debugging Applications. Microsoft Press, 2000.

Digital Library

[10]

T.J. Halloran and W.L. Scherlis, “High Quality and Open Source Software Practices,” Meeting Challenges and Surviving Success: Second Workshop Open Source Software Eng., May 2002.

[11]

B. Marick, “When Should a Test Be Automated?” Proc. 11th Int'l Software/Internet Quality Week, May 1998.

[12]

L. Crispin, T. House, and C. Wade, “The Need for Speed: Automating Acceptance Testing in an Extreme Programming Environment,” Proc. Second Int'l Conf. eXtreme Programming and Flexible Processes in Software Eng., pp. 96-104, 2001.

[13]

“WINE Daily Builds,” 2003,

[14]

“Mozilla,” 2003,

[15]

“Open WebMail,” 2003,

[16]

“Cruise Control,” 2003,

[17]

“FAST C++ Compilation— IcrediBuild by Xoreax Software,” 2003,

[18]

“Positive-g-Daily Build Product Information— Mozilla,” 2003,

[19]

“Kinook Software— Automate Software Builds with Visual Build Pro,” 2003,

[20]

N. Baran, “Load Testing Web Sites,” Dr. Dobb's J. Software Tools, vol. 26, no. 3, pp. 112, 114, 116, 118-119, Mar. 2001.

[21]

S. Ellis, D. Johnson, M. Schmit, J. Jones, S. Cooke, and K. Granroth, “Letters: Open Source Cobol; Setting the Debian Record Straight; Back to Basics; Load Testing Web Sites; Open Source Hat Tricks; KDE Insider,” Dr. Dobb's J. Software Tools, vol. 26, no. 7, pp. 10-12, July 2001.

[22]

J. Weirich, “Using Perl to Check Web Links,” Linux J., vol. 36, Apr. 1997

Digital Library

[23]

H. Berghel, “Using the WWW Test Pattern to Check HTML Client Compliance,” Computer, vol. 28, no. 9, pp. 63-65, Sept. 1995.

Digital Library

[24]

B. Marick, “Bypassing the GUI,” Software Testing and Quality Eng. Magazine, pp. 41-47, Sept. 2002.

[25]

M. Finsterwalder, “Automating Acceptance Tests for GUI Applications in an Extreme Programming Environment,” Proc. Second Int'l Conf. eXtreme Programming and Flexible Processes in Software Eng., pp. 114-117, May 2001.

[26]

L. White, H. AlMezen, and N. Alzeidi, “User-Based Testing of GUI Sequences and Their Interactions,” Proc. 12th Int'l Symp. Software Reliability Eng., pp. 54-63, 2001.

Digital Library

[27]

“JUnit, Testing Resources for Extreme Programming,”

[28]

J.H. Hicinbothom and W.W. Zachary, “A Tool for Automatically Generating Transcripts of Human-Computer Interaction,” Proc. Human Factors and Ergonomics Society 37th Ann. Meeting, p. 1042, 1993.

[29]

“Capture-Replay Tool,” 2003,

[30]

“Mercury Interactive WinRunner,” 2003,

[31]

“Abbot Java GUI Test Framework,” 2003,

[32]

“Rational Robot,” 2003,

[33]

A.M. Memon, “A Comprehensive Framework for Testing Graphical User Interfaces,” PhD thesis, Dept. of Computer Science, Univ. of Pittsburgh, July 2001.

Digital Library

[34]

“Java Test Coverage Analyzer,”

[35]

J. Su and P.R. Ritter, “Experience in Testing the Motif Interface,” IEEE Software, vol. 8, no. 2, pp. 26-33, Mar. 1991.

Digital Library

[36]

P.A. Vogel, “An Integrated General Purpose Automated Test Environment,” Proc. Int'l Symp. Software Testing and Analysis, T. Ostrand and E. Weyuker, eds., pp. 61-69, June 1993.

Digital Library

[37]

J.D. Jobson, Applied Multivariate Data Analysis Volume 1: Regression and Experimental Design. Springer, 1991

[38]

H. Sahai and M. Ageel, The Analysis of Variance: Fixed, Random and Mixed Models. Birkhauser, 2000.

Cited By

Clerissi DDenaro GMobilio MMariani L(2024)DBInputs: Exploiting Persistent Data to Improve Automated GUI TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.343900250:9(2412-2436)Online publication date: 6-Aug-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3439002
Clerissi DDenaro GMobilio MMariani L(2024)Guess the State: Exploiting Determinism to Improve GUI Exploration EfficiencyIEEE Transactions on Software Engineering10.1109/TSE.2024.336658650:4(836-853)Online publication date: 16-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3366586
Nguyen VLe B(2021)RLTCPInformation and Software Technology10.1016/j.infsof.2021.106574136:COnline publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1016/j.infsof.2021.106574
Show More Cited By

Index Terms

Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Extra-functional properties
      1. Software fault tolerance

Recommendations

What test oracle should i use for effective GUI testing?
ASE'03: Proceedings of the 18th IEEE International Conference on Automated Software Engineering

Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. In this paper we show that the test oracle used during testing also contributes ...
Designing and comparing automated test oracles for GUI-based software applications

Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle, a mechanism that determines whether a ...
Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 31, Issue 10

October 2005

111 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © 2005.

Publisher

IEEE Press

Publication History

Published: 01 October 2005

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Clerissi DDenaro GMobilio MMariani L(2024)DBInputs: Exploiting Persistent Data to Improve Automated GUI TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.343900250:9(2412-2436)Online publication date: 6-Aug-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3439002
Clerissi DDenaro GMobilio MMariani L(2024)Guess the State: Exploiting Determinism to Improve GUI Exploration EfficiencyIEEE Transactions on Software Engineering10.1109/TSE.2024.336658650:4(836-853)Online publication date: 16-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3366586
Nguyen VLe B(2021)RLTCPInformation and Software Technology10.1016/j.infsof.2021.106574136:COnline publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1016/j.infsof.2021.106574
Bajaj ASangwan O(2021)Discrete cuckoo search algorithms for test case prioritizationApplied Soft Computing10.1016/j.asoc.2021.107584110:COnline publication date: 1-Oct-2021
https://dl.acm.org/doi/10.1016/j.asoc.2021.107584
Clerissi DDenaro GMobilio MMariani LGrundy JLe Goues CLo D(2020)Plug the database & play with automatic testingProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416561(66-77)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416561
Denaro GGuglielmo LMariani LRiganelli O(2019)GUI testing in productionCompanion Proceedings of the 3rd International Conference on the Art, Science, and Engineering of Programming10.1145/3328433.3328452(1-3)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1145/3328433.3328452
Yang SWu HZhang HWang YSwaminathan CYan DRountev A(2018)Static window transition graphs for AndroidAutomated Software Engineering10.5555/3288647.328871025:4(833-873)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.5555/3288647.3288710
Adelsberger SSetzer AWalkingshaw ESabel DThiemann P(2018)Declarative GUIsProceedings of the 20th International Symposium on Principles and Practice of Declarative Programming10.1145/3236950.3236962(1-15)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3236950.3236962
Din FZamli K(2018)Fuzzy Adaptive Teaching Learning-based Optimization Strategy for GUI Functional Test Cases GenerationProceedings of the 2018 7th International Conference on Software and Computer Applications10.1145/3185089.3185148(92-96)Online publication date: 8-Feb-2018
https://dl.acm.org/doi/10.1145/3185089.3185148
Mariani LPezzè MZuddas DChaudron MCrnkovic IChechik MHarman M(2018)AugustoProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180162(280-290)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180162
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents