cispa_all_3909.pdf (390.05 kB)

SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript

Download (390.05 kB)
conference contribution
posted on 2023-11-29, 18:24 authored by Masudul Hasan Masud BhuiyanMasudul Hasan Masud Bhuiyan, Adithya Srinivas Parthasarathy, Nikos Vasilakis, Michael Pradel, Cristian-Alexandru StaicuCristian-Alexandru Staicu
Npm is the largest software ecosystem in the world, offering millions of free, reusable packages. In recent years, various security threats to packages published on npm have been reported, including vulnerabilities that affect millions of users. To continuously improve techniques for detecting vulnerabilities and mitigating attacks that exploit them, a reusable benchmark of vulnerabilities would be highly desirable. Ideally, such a benchmark should be realistic, come with executable exploits, and include fixes of vulnerabilities. Unfortunately, there currently is no such benchmark, forcing researchers to repeatedly develop their own evaluation datasets and making it difficult to compare techniques with each other. This paper presents SecBench.js, the first comprehensive benchmark suite of vulnerabilities and executable exploits for npm. The benchmark comprises 600 vulnerabilities, which cover the five most common vulnerability classes for server-side JavaScript. Each vulnerability comes with a payload that exploits the vulnerability and an oracle that validates successful exploitation. SecBench.js enables various applications, of which we explore three in this paper: (i) crosschecking SecBench.js against existing security advisories reveals 168 vulnerable versions in 19 packages that are mislabeled in the advisories; (ii) applying simple code transformations to the exploits in our suite helps identify flawed fixes of vulnerabilities; (iii) dynamically analyzing calls to common sink APIs, e.g., exec(), yields a ground truth of code locations for evaluating vulnerability detectors. Beyond providing a reusable benchmark to the community, our work identified 20 zero-day vulnerabilities, most of which are already acknowledged by practitioners.


Preferred Citation

Masudul Bhuiyan, Adithya Parthasarathy, Nikos Vasilakis, Michael Pradel and Cristian-Alexandru Staicu. SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript. In: International Conference on Software Engineering (ICSE). 2023.

Primary Research Area

  • Empirical and Behavioral Security

Name of Conference

International Conference on Software Engineering (ICSE)

Legacy Posted Date


Open Access Type

  • Green


@inproceedings{cispa_all_3909, title = "SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript", author = "Bhuiyan, Masudul Hasan Masud and Parthasarathy, Adithya Srinivas and Vasilakis, Nikos and Pradel, Michael and Staicu, Cristian-Alexandru", booktitle="{International Conference on Software Engineering (ICSE)}", year="2023", }

Usage metrics


    No categories selected


    Ref. manager