Improving ML-based Information Retrieval Software with User-driven Functional Testing and Defect Class Analysis

Download: PDF · DOI · PDF (ACM).

“Improving ML-based Information Retrieval Software with User-driven Functional Testing and Defect Class Analysis” by Junjie Zhu, Teng Long, Wei Wang, and Atif Memon. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2022. pp. 1291-1301.

Abstract

Machine Learning (ML) has become the cornerstone of information retrieval (IR) software, as it can drive better user experience by leveraging information-rich data and complex models. However, evaluating the emergent behavior of ML-based IR software can be challenging with traditional software testing approaches: when developers modify the software, they cannot often extract useful information from individual test instances; rather, they seek to holistically verify whether—and where—their modifications caused significant regressions or improvements at scale. In this paper, we introduce not only such a holistic approach to evaluate the system-level behavior of the software, but also the concept of a defect class, which represents a partition of the input space on which the ML-based software does measurably worse for an existing feature or on which the ML task is more challenging for a new feature. We leverage large volumes of functional test cases, automatically obtained, to derive these defect classes, and propose new ways to improve the IR software from an end-user's perspective. Applying our approach on a real production Search-AutoComplete system that contains a query interpretation ML component, we demonstrate that (1) our holistic metrics successfully identified two regressions and one improvement, where all 3 were independently verified with retrospective A/B experiments, (2) the automatically obtained defect classes provided actionable insights during early-stage ML development, and (3) we also detected defect classes at the finer sub-component level for which there were significant regressions, which we blocked prior to different releases.

BibTeX entry:

@inproceedings{ZhuLongWangMemonFSE2022,
   author = {Junjie Zhu and Teng Long and Wei Wang and Atif Memon},
   title = {Improving ML-based Information Retrieval Software with
     User-driven Functional Testing and Defect Class Analysis},
   booktitle = {Proceedings of the 30th ACM Joint European Software
     Engineering Conference and Symposium on the Foundations of
     Software Engineering},
   series = {ESEC/FSE 2022},
   pages = {1291--1301},
   year = {2022},
   doi = {10.1145/3540250.3558941}
}
    

Back to Atif Memon's Publications.