ICSE 2013 Tutorial: Automated Testing of GUI Applications

In conjunction with ACM/IEEE International Conference on Software Engineering (ICSE)
May 18-26, 2013, San Francisco, CA (USA)

Tutorial Overview

System testing of applications with graphical user interfaces (GUIs) such as web browsers, desktop, or mobile apps, is more complex than testing from the command line. Specialized tools are needed to generate and run test cases, models are needed to quantify behavioral coverage, and changes in the environment, such as the operating system, virtual machine or system load, as well as starting states of the executions, impact the repeatability of the outcome of tests making tests appear "flaky".

In this tutorial, we present an overview of the state of the art in GUI testing, consisting of both lectures and demonstrations on various platforms (desktop, web and mobile applications), using an open source testing tool, GUITAR. We show how to setup a system under test, how to extract models without source code, and how to then use those models to generate and replay test cases. We then present a lecture on the various factors that may cause flakiness in the execution of GUI-centric software, and hence impact the results of analyses and experiments based on such software. We end with a demonstration of a community resource for sharing GUI testing artifacts aimed at controlling these factors.

We will incorporate a combination of lectures intermingled with demonstrations. Within each lecture or demonstration period, sufficient time will be left open for questions or clarifications of key concepts.

Audience

This half-day tutorial is targeted towards a broad audience. We expect that it will be beneficial to (1) researchers who want to conduct experiments using GUI and Web based software, (2) practitioners from Industry who perform system testing, and to (3) members of the human-computer interaction (HCI) community who want to automate evaluation of tasks on graphical interfaces.

Schedule

Lecture: Introduction to GUI Testing and Models

Overview of GUI Testing
State machines and event flow graphs
Ripping (model extraction) and replaying

Demonstration: Using GUITAR to Automate the GUI Test Process
Lecture: Test Generation and Coverage

Overview of event-based coverage criteria
Test generation

Lecture: Extending GUITAR with New Test Generation Methods and Models of Interaction
Lecture: Controlling Factors

Factors that impact repeatability
Introduction to benchmarking for experimentation

Demonstration: Using COMET benchmarks
1. Downloading and configuring the benchmarks
2. Executing the benchmark subjects

Presenters

Atif M. Memon	Myra B. Cohen

Atif Memon is an Associate Professor at the Department of Computer Science, University of Maryland, where he founded and heads the Event Driven Software Lab (EDSL). He designed and developed the model-based GUI testing software GUITAR. He has published over 100 research articles on the topic of event driven systems, software testing, and software engineering. He is the founder of TESTBEDS. He also helped develop the workshop on Experimental Evaluation of Software and Systems in Computer Science (EVALUATE).	Myra Cohen is an Associate Professor in the Department of Computer Science and Engineering at the University of Nebraska-Lincoln where she is a member of the Laboratory for Empirically based Software Quality Research and Development (ESQuaReD). She is a recipient of a National Science Foundation early CAREER development award and an Air Force Office of Scientific Research young investigator program award. Her research expertise is in testing highly configurable software, GUI testing, applications of combinatorial designs, and search based software engineering.

Tutorial Materials

Coverage and Test Generation Lecture

Extending GUITAR Lecture

Controlling Factors Lecture

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant No. CNS-0855139 and CNS-0855055. CNS-1205472 and CNS-1205501.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.