Npm is the largest software ecosystem in the world, offering millions of free, reusable packages. In recent years, various security threats to packages published on npm have been reported, including vulnerabilities that affect millions of users. To continuously improve techniques for detecting vulnerabilities and mitigating attacks that exploit them, a reusable benchmark of vulnerabilities would be highly desirable. Ideally, such a benchmark should be realistic, come with executable exploits, and include fixes of vulnerabilities. Unfortunately, there currently is no such benchmark, forcing researchers to repeatedly develop their own evaluation datasets and making it difficult to compare techniques with each other. This paper presents SecBench.js, the first comprehensive benchmark suite of vulnerabilities and executable exploits for npm. The benchmark comprises 600 vulnerabilities, which cover the five most common vulnerability classes for server-side JavaScript. Each vulnerability comes with a payload that exploits the vulnerability and an oracle that validates successful exploitation. SecBench.js enables various applications, of which we explore three in this paper: (i) crosschecking SecBench.js against existing security advisories reveals 168 vulnerable versions in 19 packages that are mislabeled in the advisories; (ii) applying simple code transformations to the exploits in our suite helps identify flawed fixes of vulnerabilities; (iii) dynamically analyzing calls to common sink APIs, e.g., exec(), yields a ground truth of code locations for evaluating vulnerability detectors. Beyond providing a reusable benchmark to the community, our work identified 20 zero-day vulnerabilities, most of which are already acknowledged by practitioners.
History
Preferred Citation
Masudul Bhuiyan, Adithya Parthasarathy, Nikos Vasilakis, Michael Pradel and Cristian-Alexandru Staicu. SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript. In: International Conference on Software Engineering (ICSE). 2023.
Primary Research Area
Empirical and Behavioral Security
Name of Conference
International Conference on Software Engineering (ICSE)
Legacy Posted Date
2023-03-09
Open Access Type
Green
BibTeX
@inproceedings{cispa_all_3909,
title = "SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript",
author = "Bhuiyan, Masudul Hasan Masud and Parthasarathy, Adithya Srinivas and Vasilakis, Nikos and Pradel, Michael and Staicu, Cristian-Alexandru",
booktitle="{International Conference on Software Engineering (ICSE)}",
year="2023",
}