The widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. Although the use of GUIs continues to grow, GUI testing has remained a neglected research area. Since GUIs have characteristics that are different from those of conventional software, such as user events for input and graphical output, techniques developed to test conventional software cannot be directly applied to test GUIs. This thesis develops a unified solution to the GUI testing problem with the particular goals of automation and integration of tools and techniques used in various phases of GUI testing. These goals are accomplished by developing a GUI testing framework with a GUI model as its central component. For efficiency and scalability, a GUI is represented as a hierarchy of components, each used as a basic unit of testing. The framework also includes a test coverage evaluator, test case generator, test oracle, test executor, and regression tester. The test coverage evaluator employs hierarchical, event-based coverage criteria to automatically specify what to test in a GUI and to determine whether the test suite has adequately tested the GUI. The test case generator employs plan generation techniques from artificial intelligence to automatically generate a test suite. A test executor automatically executes all the test cases on the GUI. As test cases are being executed, a test oracle automatically determines the correctness of the GUI. The test oracle employs a model of the expected state of the GUI in terms of its constituent objects and their properties. After changes are made to a GUI, a regression tester partitions the original GUI test suite into valid test cases that represent correct input/output for the modified GUI and invalid test cases that no longer represent correct input/output. The regression tester employs a new technique to reuse some of the invalid test cases by repairing them. A cursory exploration of extending the framework to handle the new testing requirements of web-user interfaces (WUIs) is also done. The framework has been implemented and experiments have demonstrated that the developed techniques are both practical and useful.
Cited By
- Campos J, Fayollas C, Gonçalves M, Martinie C, Navarre D, Palanque P and Pinto M (2017). A More Intelligent Test Case Generation Approach through Task Models Manipulation, Proceedings of the ACM on Human-Computer Interaction, 1:EICS, (1-20), Online publication date: 30-Jun-2017.
- Campos J, Fayollas C, Martinie C, Navarre D, Palanque P and Pinto M Systematic automation of scenario-based testing of user interfaces Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, (138-148)
- Morgado I and Paiva A The iMPAcT tool Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, (876-881)
- Morgado I and Paiva A Test patterns for android mobile applications Proceedings of the 20th European Conference on Pattern Languages of Programs, (1-7)
- Pirzadeh H and Shanian S Resilient user interface level tests Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, (683-688)
- Moreira R and Paiva A Towards a pattern language for model-based GUI testing Proceedings of the 19th European Conference on Pattern Languages of Programs, (1-8)
- Anand S, Naik M, Harrold M and Yang H Automated concolic testing of smartphone apps Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, (1-11)
- Bauersfeld S and Vos T GUITest: a Java library for fully automated GUI robustness testing Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, (330-333)
- Bauersfeld S, Wappler S and Wegener J A metaheuristic approach to test sequence generation for applications with a GUI Proceedings of the Third international conference on Search based software engineering, (173-187)
- Nguyen D, Strooper P and Suess J Model-based testing of multiple GUI variants using the GUI test generator Proceedings of the 5th Workshop on Automation of Software Test, (24-30)
- Nguyen D, Strooper P and Süß J Automated functionality testing through GUIs Proceedings of the Thirty-Third Australasian Conferenc on Computer Science - Volume 102, (153-162)
- Grechanik M, Xie Q and Fu C Maintaining and evolving GUI-directed test scripts Proceedings of the 31st International Conference on Software Engineering, (408-418)
- Silva J, Saraiva J and Campos J A generic library for GUI reasoning and testing Proceedings of the 2009 ACM symposium on Applied Computing, (121-128)
- Xie Q and Memon A (2008). Using a pilot study to derive a GUI model for automated testing, ACM Transactions on Software Engineering and Methodology (TOSEM), 18:2, (1-35), Online publication date: 1-Nov-2008.
- Memon A (2008). Automatically repairing event sequence-based GUI test suites for regression testing, ACM Transactions on Software Engineering and Methodology (TOSEM), 18:2, (1-36), Online publication date: 1-Nov-2008.
- Ganov S, Killmar C, Khurshid S and Perry D Test generation for graphical user interfaces based on symbolic execution Proceedings of the 3rd international workshop on Automation of software test, (33-40)
- Brooks P and Memon A Automated gui testing guided by usage profiles Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering, (333-342)
- Gray G and Higgins C (2006). An introspective approach to marking graphical user interfaces, ACM SIGCSE Bulletin, 38:3, (43-47), Online publication date: 26-Sep-2006.
- Gray G and Higgins C An introspective approach to marking graphical user interfaces Proceedings of the 11th annual SIGCSE conference on Innovation and technology in computer science education, (43-47)
- Xie Q Developing cost-effective model-based techniques for GUI testing Proceedings of the 28th international conference on Software engineering, (997-1000)
- Memon A and Xie Q (2005). Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software, IEEE Transactions on Software Engineering, 31:10, (884-896), Online publication date: 1-Oct-2005.
- Cai K, Zhao L, Hu H and Jiang C On the Test Case Definition for GUI Testing Proceedings of the Fifth International Conference on Quality Software, (19-28)
- Kervinen A, Maunumaa M, Pääkkönen T and Katara M Model-Based testing through a GUI Proceedings of the 5th international conference on Formal Approaches to Software Testing, (16-31)
- Memon A, Nagarajan A and Xie Q (2018). Automating regression testing for evolving GUI software, Journal of Software Maintenance and Evolution: Research and Practice, 17:1, (27-64), Online publication date: 1-Jan-2005.
- Memon A and Xie Q Using Transient/Persistent Errors to Develop Automated Test Oracles for Event-Driven Software Proceedings of the 19th IEEE international conference on Automated software engineering, (186-195)
- Memon A, Banerjee I and Nagarajan A What test oracle should i use for effective GUI testing? Proceedings of the 18th IEEE International Conference on Automated Software Engineering, (164-173)
- Memon A and Soffa M (2003). Regression testing of GUIs, ACM SIGSOFT Software Engineering Notes, 28:5, (118-127), Online publication date: 1-Sep-2003.
- Kapfhammer G and Soffa M (2003). A family of test adequacy criteria for database-driven applications, ACM SIGSOFT Software Engineering Notes, 28:5, (98-107), Online publication date: 1-Sep-2003.
- Memon A and Soffa M Regression testing of GUIs Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, (118-127)
- Kapfhammer G and Soffa M A family of test adequacy criteria for database-driven applications Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, (98-107)
Recommendations
Visual testing of Graphical User Interfaces: An exploratory study towards systematic definitions and approaches
WSE '12: Proceedings of the 2012 IEEE 14th International Symposium on Web Systems Evolution (WSE)Graphical User Interface (GUI) testing literature emphasizes testing a system's functionality through its GUI, rather than testing visual aspects of the GUI itself. In this paper we introduce the notion of visual testing as a subset of GUI testing. To ...