research-article

Using a pilot study to derive a GUI model for automated testing

Authors:

Atif M MemonAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 18, Issue 2

Article No.: 7, Pages 1 - 35

https://doi.org/10.1145/1416563.1416567

Published: 12 November 2008 Publication History

Abstract

Graphical user interfaces (GUIs) are one of the most commonly used parts of today's software. Despite their ubiquity, testing GUIs for functional correctness remains an understudied area. A typical GUI gives many degrees of freedom to an end-user, leading to an enormous input event interaction space that needs to be tested. GUI test designers generate and execute test cases (modeled as sequences of user events) to traverse its parts; targeting a subspace in order to maximize fault detection is a nontrivial task. In this vein, in previous work, we used informal GUI code examination and personal intuition to develop an event-interaction graph (EIG). In this article we empirically derive the EIG model via a pilot study, and the resulting EIG validates our intuition used in previous work; the empirical derivation process also allows for model evolution as our understanding of GUI faults improves. Results of the pilot study show that events interact in complex ways; a GUI's response to an event may vary depending on the context established by preceding events and their execution order. The EIG model helps testers to understand the nature of interactions between GUI events when executed in test cases and why certain events detect faults, so that they can better traverse the event space. New test adequacy criteria are defined for the EIG; new algorithms use these criteria and EIG to systematically generate test cases that are shown to be effective on four fielded open-source applications.

References

[1]

Andrews, J. H., Briand, L. C., and Labiche, Y. 2005. Is mutation an appropriate tool for testing experiments? In ICSE '05: Proceedings of the 27th International Conference on Software Engineering. ACM Press, New York, NY, 402--411.

Digital Library

[2]

Auguston, M., Michael, J. B., and Shing, M.-T. 2005. Environment behavior models for scenario generation and testing automation. In A-MOST '05: Proceedings of the 1st International Workshop on Advances in Model-Based Testing. ACM Press, New York, NY, 1--6.

Digital Library

[3]

Bernhard, P. J. 1994. A reduced test suite for protocol conformance testing. ACM Trans. Softw. Eng. Methodol. 3, 3 (July), 201--220.

Digital Library

[4]

Chow, T. S. 1978. Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. SE-4, 3, 178--187.

Digital Library

[5]

Clarke, J. M. 1998. Automated test generation from a behavioral model. In Proceedings of the Pacific Northwest Software Quality Conference (Pnsqc/Pacific Agenda, Portland, OR).

[6]

Esmelioglu, S. and Apfelbaum, L. 1997. Automated test generation, execution, and reporting. In Proceedings of the Pacific Northwest Software Quality Conference (Pnsqc/Pacific Agenda, Portland, OR). 127--142.

[7]

Harrold, M. J., Offut, A. J., and Tewary, K. 1997. An approach to fault modelling and fault seeding using the program dependence graph. J. Syst. Softw. 36, 3 (Mar.), 273--296.

Digital Library

[8]

Hicinbothom, J. H. and Zachary, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations, vol. 2. Human Factors and Ergonomics Society, Santa Monica, CA, 1042.

[9]

Howe, A., von Mayrhauser, A., and Mraz, R. T. 1997. Test case generation as an AI planning problem. Automat. Softw. Eng. 4, 77--106.

Digital Library

[10]

Imanian, J. A. 2005. Automatic test case generation for reactive software systems based on environment models. Ph.D. dissertation. Naval Postgraduate School, Monterey, CA.

[11]

Jorgensen, P. C. and Erickson, C. 1994. Object-oriented integration testing. Commun. ACM 37, 9, 30--38.

Digital Library

[12]

Kasik, D. J. and George, H. G. 1996. Toward automatic generation of novice user test scripts. In Proceedings of the Conference on Human Factors in Computing Systems: Common Ground. ACM Press, New York, NY, 244--251.

Digital Library

[13]

Leow, W. K., Khoo, S. C., and Sun, Y. 2004. Automated generation of test programs from closed specifications of classes and test cases. In ICSE '04: Proceedings of the 26th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 96--105.

Digital Library

[14]

Maurer, P. M. 1990. Generating test data with enhanced context-free grammars. IEEE Softw. 7, 4 (July), 50--55.

Digital Library

[15]

Memon, A., Banerjee, I., and Nagarajan, A. 2003. GUI Ripping: Reverse engineering of graphical user interfaces for testing. In WCRE '03: Proceedings of the 10th Working Conference on Reverse Engineering. IEEE Computer Society Press, Los Alamitos, CA, 260--269.

Digital Library

[16]

Memon, A. M. 2001. A comprehensive framework for testing graphical user interfaces. Ph.D. dissertation. Department of Computer Science, University of Pittsburgh, Pittsburgh, PA.

Digital Library

[17]

Memon, A. M. 2002. GUI testing: Pitfalls and process. IEEE Comput. 35, 8 (Aug.), 90--91.

Digital Library

[18]

Memon, A. M. 2003. Advances in GUI testing. In Advances in Computers, 58, M. V. Zelkowitz, Ed. Elsevier, Amsterdam, The Netherlands, 150--203.

[19]

Memon, A. M., Nagarajan, A., and Xie, Q. 2005. Automating regression testing for evolving GUI software. J. Softw. Maint. 17, 1, 27--64.

Digital Library

[20]

Memon, A. M., Pollack, M. E., and Soffa, M. L. 1999. Using a goal-driven approach to generate test cases for GUIs. In ICSE '99: Proceedings of the 21st International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 257--266.

Digital Library

[21]

Memon, A. M., Pollack, M. E., and Soffa, M. L. 2000. Automated test oracles for GUIs. In SIGSOFT '00/FSE-8: Proceedings of the 8th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 30--39.

Digital Library

[22]

Memon, A. M., Pollack, M. E., and Soffa, M. L. 2001a. Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27, 2 (Feb.), 144--155.

Digital Library

[23]

Memon, A. M. and Soffa, M. L. 2003. Regression testing of GUIs. In ESEC/FSE-11: Proceedings of the 9th European Software Engineering Conference Held Jointly with the 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 118--127.

Digital Library

[24]

Memon, A. M., Soffa, M. L., and Pollack, M. E. 2001b. Coverage criteria for GUI testing. In ESEC/FSE-9: Proceedings of the 8th European Software Engineering Conference Held Jointly with the 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY, 256--267.

Digital Library

[25]

Memon, A. M. and Xie, Q. 2005. Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31, 10 (Oct.), 884--896.

Digital Library

[26]

Offutt, A. J. and Hayes, J. H. 1996. A semantic model of program faults. In ISSTA '96: Proceedings of the 1996 ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York, NY, 195--200.

Digital Library

[27]

Rapps, S. and Weyuker, E. J. 1982. Data flow analysis techniques for test data selection. In ICSE '82: Proceedings of the 6th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 272--278.

Digital Library

[28]

Scheetz, M., von Mayrhauser, A., France, R., Dahlman, E., and Howe, A. E. 1999. Generating test cases from an OO model with an AI planning system. In Proceedings of the 10th International Symposium on Software Reliability Engineering. IEEE Computer Society Press, Los Alamitos, CA, 250--259.

Digital Library

[29]

Shehady, R. K. and Siewiorek, D. P. 1997. A method to automate user interface testing using variable finite state machines. In Proceedings of the 27th Annual International Symposium on Fault-Tolerant Computing (FTCS'97). IEEE Computer Society Press, Los Alamitos, CA, 80--88.

Digital Library

[30]

Spacco, J., Hovemeyer, D., and Pugh, W. 2006. Tracking defect warnings across versions. In MSR '06: Proceedings of the 2006 International Workshop on Mining Software Repositories. ACM Press, New York, NY, 133--136.

Digital Library

[31]

von Mayrhauser, A. and Crawford-Hines, S. 1993. Automated testing support for a robot tape library. In Proceedings of the 4th International Software Reliability Engineering Conference. IEEE Computer Society Press, Los Alamitos, CA, 6--14.

[32]

von Mayrhauser, A., Mraz, R. T., and Walls, J. 1994. Domain based regression testing. In Proceedings of the International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 26--35.

Digital Library

[33]

White, L. and Almezen, H. 2000. Generating test cases for GUI responsibilities using complete interaction sequences. In ISSRE '00: Proceedings of the 11th International Symposium on Software Reliability Engineering. IEEE Computer Society Press, Los Alamitos, CA, 110.

Digital Library

[34]

Whittaker, J. A. 1992. Markov chain techniques for software testing and reliability analysis. Ph.D. dissertation. University of Tennessee, Knoxville, TN.

Digital Library

[35]

Whittaker, J. A. and Thomason, M. G. 1994. A Markov chain model for statistical software testing. IEEE Trans. Softw. Eng. 20, 10, 812--824.

Digital Library

[36]

Woit, D. 1998. Conditional-event usage testing. In CASCON '98: Proceedings of the 1998 Conference of the Centre for Advanced Studies on Collaborative Research. IBM Press, Indianapolis, IN, 23.

Digital Library

[37]

Woit, D. M. 1993. Specifying operational profiles for modules. In ISSTA '93: Proceedings of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM Press, New York, NY, 2--10.

Digital Library

[38]

Xie, Q. and Memon, A. M. 2005. Rapid “crash testing” for continuously evolving GUI-based software applications. In ICSM '05: Proceedings of the 21st IEEE International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 473--482.

Digital Library

[39]

Xie, Q. and Memon, A. M. 2006. Automated model-based testing of community-driven open source GUI applications. In ICSM '06: Proceedings of the 22nd IEEE International Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 145--154.

Digital Library

[40]

Xie, T., Marinov, D., Schulte, W., and Notkin, D. 2005. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 05). Spinger, Berlin, Germany, 365--381.

Digital Library

[41]

Zeller, A. 2002. Isolating cause-effect chains from computer programs. ACM SIGSOFT Softw. Eng. Notes 27, 6, 1--10.

Digital Library

Cited By

Coutinho MMarques LSantos ADahia MFrança Cde Souza Santos RAdams BZimmermann TOzkaya ILin DZhang J(2024)The Role of Generative AI in Software Development Productivity: A Pilot Case StudyProceedings of the 1st ACM International Conference on AI-Powered Software10.1145/3664646.3664773(131-138)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3664646.3664773
Kumar SNitin Yadav M(2023)GUI Testing Using Random Event-Based Test Cases2023 12th International Conference on System Modeling & Advancement in Research Trends (SMART)10.1109/SMART59791.2023.10428625(512-520)Online publication date: 22-Dec-2023
https://doi.org/10.1109/SMART59791.2023.10428625
Santiago VÖzcan EBalera J(2022)Many-objective test case generation for graphical user interface applications via search-based and model-based testingExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118075208:COnline publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.eswa.2022.118075
Show More Cited By

Index Terms

Using a pilot study to derive a GUI model for automated testing
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Software management
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Designing and comparing automated test oracles for GUI-based software applications

Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle, a mechanism that determines whether a ...
An Extensible Heuristic-Based Framework for GUI Test Case Maintenance
ICSTW '09: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops

Graphical user interfaces (GUIs) make up a large portion of the code comprising many modern software applications. However, GUI testing differs significantly from testing of traditional software. One respect in which this is true is test case ...
GUI Interaction Testing: Incorporating Event Context

Graphical user interfaces (GUIs), due to their event-driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing, it is important to “adequately cover” this interaction space. In this paper, we ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 18, Issue 2

November 2008

130 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/1416563

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2008

Accepted: 01 October 2007

Revised: 01 July 2007

Received: 01 August 2006

Published in TOSEM Volume 18, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

67
Total Citations
View Citations
1,520
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)5

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Coutinho MMarques LSantos ADahia MFrança Cde Souza Santos RAdams BZimmermann TOzkaya ILin DZhang J(2024)The Role of Generative AI in Software Development Productivity: A Pilot Case StudyProceedings of the 1st ACM International Conference on AI-Powered Software10.1145/3664646.3664773(131-138)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3664646.3664773
Kumar SNitin Yadav M(2023)GUI Testing Using Random Event-Based Test Cases2023 12th International Conference on System Modeling & Advancement in Research Trends (SMART)10.1109/SMART59791.2023.10428625(512-520)Online publication date: 22-Dec-2023
https://doi.org/10.1109/SMART59791.2023.10428625
Santiago VÖzcan EBalera J(2022)Many-objective test case generation for graphical user interface applications via search-based and model-based testingExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118075208:COnline publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.eswa.2022.118075
Kilincceker OSilistre ABelli FChallenger M(2021)Model-Based Ideal Testing of GUI Programs–Approach and Case StudiesIEEE Access10.1109/ACCESS.2021.30775189(68966-68984)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3077518
Nass MAlégroth EFeldt R(2021)Why many challenges with GUI test automation (will) remainInformation and Software Technology10.1016/j.infsof.2021.106625138(106625)Online publication date: Oct-2021
https://doi.org/10.1016/j.infsof.2021.106625
Silistre AKilincceker OBelli FChallenger MKardas G(2020)Models in Graphical User Interface Testing: Study Design2020 Turkish National Software Engineering Symposium (UYMS)10.1109/UYMS50627.2020.9247072(1-6)Online publication date: 7-Oct-2020
https://doi.org/10.1109/UYMS50627.2020.9247072
Pareek PChande S(2020)Grey Box Approach for Mobile Application TestingRising Threats in Expert Applications and Solutions10.1007/978-981-15-6014-9_47(411-419)Online publication date: 2-Oct-2020
https://doi.org/10.1007/978-981-15-6014-9_47
Seo YHuh J(2019)GUI-based software modularization through module clustering in edge computing based IoT environmentsJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-019-01455-3Online publication date: 6-Sep-2019
https://doi.org/10.1007/s12652-019-01455-3
khan Miftikhar SIqbal MSherin S(2018)Empirical studies omit reporting necessary detailsComputer Standards & Interfaces10.1016/j.csi.2017.08.00255:C(156-170)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.csi.2017.08.002
Yang SWu HZhang HWang YSwaminathan CYan DRountev A(2018)Static window transition graphs for AndroidAutomated Software Engineering10.1007/s10515-018-0237-625:4(833-873)Online publication date: 16-Jun-2018
https://doi.org/10.1007/s10515-018-0237-6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents