CISPA
Browse
cispa_all_3573.pdf (596.84 kB)

Reproducibility and Replicability of Web Measurement Studies

Download (596.84 kB)
conference contribution
posted on 2023-11-29, 18:19 authored by Nurullah Demir, Matteo Große-Kampmann, Tobias Urban, Christian Wresnegger, Thorsten HolzThorsten Holz, Norbert Pohlmann
Web measurement studies can shed light on not yet fully understood phenomena and thus are essential for analyzing how the modern Web works. This often requires building new and adjusting existing crawling setups, which has led to a wide variety of analysis tools for different (but related) aspects. If these efforts are not sufficiently documented, the reproducibility and replicability of the measurements may suffer - two properties that are crucial to sustainable research. In this paper, we survey 117 recent research papers to derive best practices for Web-based measurement studies and specify criteria that need to be met in practice. When applying these criteria to the surveyed papers, we find that the experimental setup and other aspects essential to reproducing and replicating results are often missing. We underline the criticality of this finding by performing a large-scale Web measurement study on 4.5 million pages with 24 different measurement setups to demonstrate the influence of the individual criteria. Our experiments show that slight differences in the experimental setup directly affect the overall results and must be documented accurately and carefully.

History

Preferred Citation

Nurullah Demir, Matteo Große-Kampmann, Tobias Urban, Christian Wresnegger, Thorsten Holz and Norbert Pohlmann. Reproducibility and Replicability of Web Measurement Studies. In: The Web Conference (WWW). 2022.

Primary Research Area

  • Threat Detection and Defenses

Name of Conference

The Web Conference (WWW)

Legacy Posted Date

2022-02-09

Open Access Type

  • Unknown

BibTeX

@inproceedings{cispa_all_3573, title = "Reproducibility and Replicability of Web Measurement Studies", author = "Demir, Nurullah and Große-Kampmann, Matteo and Urban, Tobias and Wresnegger, Christian and Holz, Thorsten and Pohlmann, Norbert", booktitle="{The Web Conference (WWW)}", year="2022", }

Usage metrics

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC