Repurposing Automated Testing for ML Software Development and DeploymentML-powered features — and increasingly LLM-powered ones — behave non-deterministically and emerge from training rather than specification, so the classical assumption of a testable spec breaks down. Traditional testing during code development and metrics collected during model development fail to give an accurate view of user-level bugs, and online A/B metrics don't directly expose them either. The talk shows how large volumes of system test cases, mined for defect classes (partitions of the input space where ML-based software measurably underperforms), expose these bugs early and drive rapid improvement.31 Mar–4 Apr 2025 · 18th IEEE International Conference on Software Testing, Verification and Validation (ICST 2025), Naples, Italy · researchr.org
MachineLearning × SoftwareTesting @ Apple MapsML systems have two design characteristics — their ever-improving nature and learning-based emergent behavior — that create a moving target for functional regression testing. Unlike classical software, there is no fixed specification to test against, and expected outputs drift as the model learns from new data. The keynote presents a general methodology for automatically authoring and maintaining tests at planetary scale: using production data volume to refresh test inputs and expected outputs, data perturbation to achieve coverage adequacy, and clustering to triage failure patterns indicative of real bugs.May 2022 · 44th International Conference on Software Engineering (ICSE 2022) — SEIP Track · researchr.org
Acknowledging the Elephant in the Room: Brittle Test CasesFlaky tests — those that non-deterministically pass or fail under seemingly identical conditions — are a widespread but under-acknowledged problem in regression testing. The keynote names the elephant in the room and surveys the causes, consequences, and approaches for detecting and repairing flaky behavior. The topic has only grown in importance since: continuous-integration pipelines and ML-driven features amplify the cost of each flaky signal, and the techniques raised here foreshadow the flaky-test ranking and ML-based testing work that followed.17 Apr 2012 · Regression 2012, Montreal
Invited Talks
Get the Most out of Test Automation at Scale: Listen to your DataData-driven strategies for scaling test automation: using signals from millions of test runs to decide what to test, when to test, which failures matter, and which tests to repair. Lessons from Google-scale continuous testing, where classical test-selection and prioritization techniques had to be rethought once the test infrastructure started generating orders-of-magnitude more data than engineers could triage manually.2017 · Netflix, Los Gatos, CA · YouTube
How Flaky Tests in Continuous IntegrationFlaky tests — tests that non-deterministically pass or fail — are a widespread problem in continuous-integration systems at scale, where even a small flake rate can gum up thousands of builds per day. The talk examines how to detect, rank, and repair flaky test scripts, foreshadowing the Google-scale continuous testing work published the following year.2016 · Google Test Automation Conference (GTAC 2016), Sunnyvale, CA · YouTube
Recent Advances in Software TestingA survey of recent advances in software testing: model-based testing, GUI ripping, test automation at scale, flakiness detection and repair, and the emerging challenges posed by machine-learning-based features in modern software.2016 · Apple, Sunnyvale, CA
Model-Based Dynamic Analysis: Pushing the Envelope on Automated Software Quality AssuranceA synthesis of model-based dynamic analysis for software QA: automatically reverse-engineering behavioral models from running software, generating test cases from those models, and using run-time feedback to adapt the test suite. The talk connects event-flow modeling and GUI Ripping work to broader software quality assurance practice.Given at:
2015 · Google, Mountain View, CA
2015 · Yahoo Labs, Palo Alto, CA
2015 · Dept. of Electrical & Computer Engineering, University of Miami
Situation-Driven TestingPanel contribution arguing that modern software testing must become situation-driven — adaptively tuning what, when, and how to test based on run-time context, usage patterns, and the surrounding software ecosystem — rather than following a static test plan.2015 · ICST 2015 Grand Challenge Panel
The First Decade of GUI Ripping: Extensions, Applications, and Broader ImpactsA retrospective on GUI Ripping — reverse-engineering a workflow model from the running GUI — a decade after its introduction. Covers how the technique scaled to millions-of-test-case experiments, ported across platforms (Java, iOS, Android, Windows, web), and shifted GUI testing from a largely manual practice to an automation-first field.Given at:
2014 · Dept. of Computer Science, Stanford University
2014 · Dept. of Computer Science, Virginia Tech
2014 · Groupon, Inc.
2014 · Dept. of Computer Science, University of Miami
2013 · 2003 Working Conference on Reverse Engineering, Koblenz, Germany
Implementing Continuous Integration and Testing: Issues, Challenges, and SolutionsPractical issues in rolling out continuous-integration and continuous-testing pipelines in a large organization: balancing test-suite size against build-time budgets, handling flaky tests, selecting which tests to run on which changes, and reporting results in ways developers will actually act on.2014 · McGraw Hill Financial Services
GUITAR: Kick-starting Model-Based Test Automation for User-Interactive Event-Driven SoftwareGUITAR is a GUI testing tool whose plug-in architecture supports many testing techniques — model-based, capture/replay, manually scripted — within a single, extensible framework. Software developers and QA engineers can compose new toolchains and workflows and plug in measurement tools to drive end-to-end GUI testing.22 May 2013 · VMware Inc., Palo Alto, CA
Automated Testing of GUI Applications: Models, Tools, and Controlling FlakinessSystem testing of GUI-driven applications (web, desktop, mobile) requires models for behavioral coverage, specialized tools for test generation and execution, and awareness that environmental factors (OS, VM, system load, starting state) make tests appear flaky. The talk walks through how to set up a system under test, extract models without source code, generate and replay test cases with the open-source GUITAR tool, and share testing artifacts to control flakiness.21 May 2013 · Apple Inc., Cupertino, CA
COMET: A Web Infrastructure for Research and Experimentation in User-Interactive Event-Driven TestingCOMET is an NSF-funded web infrastructure for community-based testing and experimentation on user-interactive, event-driven software (such as GUIs and web applications). It provides a shared platform where researchers can register subject applications, test artifacts, and tools, enabling large-scale, cross-institutional experiments on test generation, oracle design, and coverage evaluation.Given at:
20 Dec 2012 · Software Institute, Nanjing University, China
17 Dec 2012 · National Institute of Advanced Industrial Science and Technology (AIST), Japan
13 Dec 2012 · Dept. of Informatics Engineering, University of Porto, Portugal
Which Test Oracle Should I Use for Effective GUI Testing?Test effectiveness and cost depend not just on the test cases but equally on the test oracle: the mechanism that decides whether an execution was correct. The talk compares 11 types of oracles across 660,000 test runs on four systems, showing that detailed oracle information with more complex procedures finds faults earlier and with fewer test cases, albeit at higher per-test cost.22 May 2012 · 20th CREST Open Workshop “The Oracle Problem for Automated Software Testing”
Adaptive Test-Case Generation for Event-Driven Software ApplicationsA fully automatic, feedback-driven technique for generating GUI test cases: a seed suite is run, the runtime effect of each event on all other events pinpoints “event-semantic interaction” relationships, and these drive the generation of new, more effective test cases. Studies on eight applications show that the feedback-based approach significantly outperforms baseline techniques and detects more faults than code-, event-, or interaction-coverage equivalents.Given at:
30 Sep 2011 · University of Lugano, Switzerland
9 Nov 2009 · IBM T. J. Watson Research Center
Automated Model-Based Testing of Event-Driven Software ApplicationsA consolidated, scalable event-flow model for event-driven applications (such as GUIs) that unifies previously separate models for test-case generation, oracle creation, coverage evaluation, and regression testing. The model is semi-automatically reverse-engineered from an implementation and drives end-to-end testing through event-space exploration strategies demonstrated on real applications.Given at:
16 Dec 2009 · Tata Consultancy Services (TCS), Chennai, India
6 Nov 2009 · Tata Research Development and Design Centre (TRDDC), Pune, India
24 Sep 2008 · Institute of Software, Chinese Academy of Sciences (ISCAS), China
11 Sep 2008 · Sichuan University, Chengdu, China
24 Jun 2008 · IEEE Computer Society of Pakistan
11 May 2007 · Accenture Technology Labs, Chicago
Software Quality and GUITARHow software quality assurance is strengthened by automated GUI testing, and a walkthrough of GUITAR — a GUI testing tool whose plug-in architecture supports many testing techniques within a single, extensible framework. The talk demonstrates how teams can set up GUITAR on their own systems and plug in measurement tools to drive end-to-end GUI testing.Given at:
11 Sep 2008 · Sichuan University, Chengdu, China
1 Dec 2003 · National Institutes of Health (NIH)
New Trends in Software TestingOverview of emerging directions in software testing: model-based test generation, event-driven and GUI testing, test-oracle design, continuous testing, and the integration of experimentation infrastructure to let researchers and practitioners compare techniques on common benchmarks.21 Jun 2008 · National University of Emerging and Computing Sciences (NUCES)
Testing Event-Driven SystemsTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.5 Mar 2008 · Information Technology Laboratory (ITL), National Institute of Standards & Technology (NIST)
Automated Model-Based Testing of Web ApplicationsModel-based test automation applied to web applications: reverse-engineering a workflow model of the application and using it to generate, execute, and maintain system-level test cases. The talk shows how techniques developed for GUI testing (event-flow modeling, GUI ripping, test-oracle design) extend naturally to the browser.2008 · Google Test Automation Conference (GTAC 2008) · YouTube
Skoll: Distributed Continuous Quality Assurance (DCQA)Skoll is an environment for distributed, continuous quality assurance that scales to highly configurable systems by leveraging the computing resources of user and developer communities around the world. Case studies on the Mozilla browser and the ACE+TAO communication package show Skoll rapidly identifying problems that conventional QA processes had missed or taken much longer to find.23–24 Aug 2007 · Google Test Automation Conference (GTAC 2007) · YouTube
Testing Event-Driven Software Applications: Issues, Challenges, and SolutionsTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.17 Mar 2006 · IEEE Computer Society Kitchener-Waterloo Chapter & Dept. of Electrical & Computer Engineering, University of Waterloo
Testing Event-Driven SoftwareTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.Given at:
11 Feb 2005 · Dept. of Computer Science, Drexel University
12 Mar 2004 · Microsoft Research
4 Mar 2004 · Reliability Engineering Dept., University of Maryland
3 Mar 2004 · Dept. of Information & Software Engineering, George Mason University
19 Feb 2004 · Fraunhofer IESE, Kaiserslautern, Germany
Enhancing Testing Techniques for Event-Driven Software ApplicationsInvited Talk22–26 Aug 2004 · 2004 University of Washington & Microsoft Research Summer Institute — Trends in Testing: Theory, Techniques and Tools
Event-Driven Software TestingTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.13 Jul 2004 · NASA Goddard Space Flight Center (GSFC), Technology Education & Assessment Seminars (TEAS)
Techniques and Tools for Testing Event-Driven SoftwareTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.19 Mar 2004 · TECH 2004
Event-Based Software TestingTesting techniques for event-driven software — applications like GUIs, web apps, and network protocols that take sequences of events as input and change state in response. The talk covers model-based test generation, test oracles, coverage criteria, and regression testing specific to the event-driven paradigm.Given at:
12 Jun 2003 · Avaya Labs Research
14 May 2003 · Titan Corporation
Testing Graphical User Interface SoftwareAn early talk on model-based GUI testing: using a hierarchical representation of the GUI to specify coverage criteria, generate test cases, and automate oracle checking. The approach decomposes the GUI into components and uses event-flow graphs to drive systematic test generation.23 Apr 2002 · NASA Goddard Tech Summit
A Comprehensive Framework for Testing Graphical User InterfacesA unified framework for testing graphical user interfaces, built around a hierarchical GUI model that serves as the central component for coverage evaluation, test-case generation (via AI planning), oracle checking, execution, and regression testing. The framework repairs invalid regression test cases after GUI changes and demonstrates on real applications that model-based GUI testing can be both practical and useful.Given at:
8 Jan 2002 · National Institute of Standards and Technology (NIST), Gaithersburg, MD
10 Apr 2001 · Dept. of Computer Science, University of Southern California
9 Apr 2001 · Dept. of Information & Computer Science, University of California, Irvine
2 Apr 2001 · College of Computing, Georgia Institute of Technology
27 Mar 2001 · Dept. of Electrical & Computer Engineering, University of Texas at Austin
20 Mar 2001 · Dept. of Computer Science & Engineering, Pennsylvania State University
28 Feb 2001 · Dept. of Computer Science, University of Maryland
23 Feb 2001 · Lucent Technologies, Columbus
20 Feb 2001 · Dept. of Computer & Information Sciences, University of Delaware
8 Feb 2001 · Dept. of Computer & Information Science, Ohio State University
22 Jan 2001 · IBM T. J. Watson Research Center
Coverage Criteria for GUI TestingNew coverage criteria for GUI testing based on events and event sequences, exploiting the GUI's hierarchical structure to identify important sequences to test. An event-flow graph captures intra-component interactions and an integration tree captures inter-component ones; a case study shows a strong correlation between event-based coverage of the GUI and statement coverage of the underlying code.17 Sep 2001 · Fraunhofer Center for Experimental Software Engineering, College Park, MD