How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency Practices

Transparent research reporting is crucial to understanding and assessing research, its results and validity, and for fostering replication. While other research fields investigated reporting and transparency practices, similar meta-research is missing for the usable privacy and security (UPS) community, which combines security, privacy, and human research. To gain insights into current research transparency practices and their development in the UPS community, we analyzed 200 UPS publications from twelve venues (including USENIX Security, IEEE S&P, CCS, SOUPS, and CHI) from 2018 to 2023. Additionally, we evaluated those venues’ 81 calls for papers (CfPs) and 20 calls for artifacts (CfAs). We find that most papers report on many of 52 analyzed transparency criteria, but none achieve full transparency. Moreover, we uncover several areas that need improvements: essential artifacts like questionnaires are frequently missing and hinder replication, some information is reported inconsistently, and dead links further reduce availability. Our regression analysis indicates that paper length and the number of studies described in a paper impact reporting transparency, while we observed no effect of publication year and artifact evaluation (AE). Finally, we provide recommendations for authors, venues, and PC chairs to improve research transparency practices and suggest transparency guidelines.