cispa_all_3737.pdf (1.82 MB)

Where to Recruit for Security Development Studies: Comparing Six Software Developer Samples

Download (1.82 MB)
conference contribution
posted on 2023-11-29, 18:21 authored by Harjot Kaur, Sabrina AmftSabrina Amft, Daniel Votipka, Yasemin Acar, Sascha FahlSascha Fahl
Studying developers is an important aspect of usable security and privacy research. In particular, studying security development challenges such as the usability of security APIs, the secure use of information sources during development or the effectiveness of IDE security plugins raised interest in recent years. However, recruiting skilled participants with software development experience is particularly challenging, and it is often not clear what security researchers can expect from certain participant samples, which can make research results hard to compare and interpret. Hence, in this work, we study for the first time opportunities and challenges of different platforms to recruit participants with software development experience for security development studies. First, we identify popular recruitment platforms in 59 papers. Then, we conduct a comparative online study with 706 participants based on self-reported software development experience across six recruitment platforms. Using an online questionnaire, we investigate participants’ programming and security experiences, skills and knowledge. We find that participants across all samples report rich general software development and security experience, skills, and knowledge. Based on our results, we recommend developer recruitment from Upwork for practical coding studies and Amazon MTurk along with a pre-screening survey to reduce additional noise for larger studies. Both of these, along with Freelancer, are also recommended for security studies. We conclude the paper by discussing the impact of our results on future security development studies.


Preferred Citation

Harjot Kaur, Sabrina Amft, Daniel Votipka, Yasemin Acar and Sascha Fahl. Where to Recruit for Security Development Studies: Comparing Six Software Developer Samples. In: Usenix Security Symposium (USENIX-Security). 2022.

Primary Research Area

  • Empirical and Behavioral Security

Name of Conference

Usenix Security Symposium (USENIX-Security)

Legacy Posted Date


Open Access Type

  • Green


@inproceedings{cispa_all_3737, title = "Where to Recruit for Security Development Studies: Comparing Six Software Developer Samples", author = "Kaur, Harjot and Amft, Sabrina and Votipka, Daniel and Acar, Yasemin and Fahl, Sascha", booktitle="{Usenix Security Symposium (USENIX-Security)}", year="2022", }

Usage metrics


    No categories selected


    Ref. manager