CISPA
Browse

Tests4Py: A Benchmark for System Testing

Download (634.2 kB)
conference contribution
posted on 2024-07-25, 12:37 authored by Marius SmytzekMarius Smytzek, Martin Eberlein, Batuhan SerceBatuhan Serce, Lars Grunske, Andreas ZellerAndreas Zeller
Benchmarks are among the main drivers of progress in software engineering research. However, many current benchmarks are limited by inadequate system oracles and sparse unit tests. Our Tests4Py benchmark, derived from the BugsInPy benchmark, addresses these limitations. It includes 73 bugs from seven real-world Python applications and six bugs from example programs. Each subject in Tests4Py is equipped with an oracle for verifying functional correctness and supports both system and unit test generation. This allows for comprehensive qualitative studies and extensive evaluations, making Tests4Py a cutting-edge benchmark for research in test generation, debugging, and automatic program repair.

History

Primary Research Area

  • Threat Detection and Defenses

Name of Conference

European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)

Page Range

557-561

Publisher

Association for Computing Machinery (ACM)

Open Access Type

  • Green

BibTeX

@conference{Smytzek:Eberlein:Serce:Grunske:Zeller:2024, title = "Tests4Py: A Benchmark for System Testing", author = "Smytzek, Marius" AND "Eberlein, Martin" AND "Serce, Batuhan" AND "Grunske, Lars" AND "Zeller, Andreas", year = 2024, month = 7, pages = "557--561", publisher = "Association for Computing Machinery (ACM)", doi = "10.1145/3663529.3663798" }

Usage metrics

    Categories

    No categories selected

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC