CISPA
Browse
cispa_all_3474.pdf (2.08 MB)

When Life Gives You Oranges: Detecting and Diagnosing Intermittent Job Failures at Mozilla

Download (2.08 MB)
conference contribution
posted on 2023-11-29, 18:17 authored by Johannes LampelJohannes Lampel, Sascha Just, Sven Apel, Andreas ZellerAndreas Zeller
Continuous delivery of cloud systems requires constant running of jobs (build processes, tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent failures - non-deterministic, false alarms that do not result from a bug in the software or job specification, but rather from issues in the underlying infrastructure. At Mozilla, such intermittent failures are called oranges as a reference to the color of the build status indicator. As such intermittent failures disrupt CI and lead to failures, they erode the developers' trust in the jobs. We present a novel approach that automatically classifies failing jobs to determine whether job execution failures arise from an actual software bug or were caused by flakiness in the job (e.g., test) or the underlying infrastructure. For this purpose, we train classification models using job telemetry data to diagnose failure patterns involving features such as runtime, cpu load, operating system version, or specific platform with high precision. In an evaluation on a set of Mozilla CI jobs, our approach achieves precision scores of 73%, on average, across all data sets with some test suites achieving precision scores good enough for fully automated classification (i.e., precision scores of up to 100%), and recall scores of 82% on average (up to 94%).

History

Preferred Citation

Johannes Lampel, Sascha Just, Sven Apel and Andreas Zeller. When Life Gives You Oranges: Detecting and Diagnosing Intermittent Job Failures at Mozilla. In: European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 2021.

Primary Research Area

  • Empirical and Behavioral Security

Name of Conference

European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)

Legacy Posted Date

2021-09-22

Open Access Type

  • Unknown

BibTeX

@inproceedings{cispa_all_3474, title = "When Life Gives You Oranges: Detecting and Diagnosing Intermittent Job Failures at Mozilla", author = "Lampel, Johannes and Just, Sascha and Apel, Sven and Zeller, Andreas", booktitle="{European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)}", year="2021", }

Usage metrics

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC