A preprint review of replication work in usable security and privacy found no exact replications among the 24 papers that met its criteria. The researchers classified all 24 as conceptual replications—follow-up studies that revisited earlier work with changes to the study design, method or analysis.
The result is a description of the studies in this review, not a scorecard of whether earlier findings survived. The project examined how venues define replication, what kinds of papers appear and why authors undertake them; it did not pool the underlying user-study results into a replication success rate.
The review’s boundaries
The team read Calls for Papers from 13 venues, searched publications from 2016 through 2025 using replication-related terms and screened 1,232 initial records. After title, abstract and full-text checks, 1,196 records were excluded, leaving 24 replication papers involving studies with people directly as participants; work based only on secondary user data was left out.
Only about half of the 13 venues explicitly encouraged replications in their Calls for Papers. SOUPS was the only venue that gave concrete instructions on using a replication to confirm, question or clarify earlier results and on describing methodological differences.
Most of the identified papers appeared at SOUPS, with 11, and CHI, with six. Those are counts within the review’s selected corpus, not an estimate of the field’s overall publication rate.
Most replications changed the setup
Three researchers assigned the 24 studies to eight groups using an adapted framework that tracks changes in the research domain, method and analysis—in other words, where the question was studied, how evidence was gathered and how it was examined.
Nineteen papers changed the domain and 19 changed the method. Seven changed the analysis; all seven also changed the method. Fourteen changed both domain and method.
Under the review’s framework, that pattern meant every paper was labeled conceptual and about two-thirds changed more than one aspect of the original study. None was classified as an exact replication.
Why authors repeated studies
The author survey drew 25 participants, while 17 emails were undeliverable. Two researchers coded the responses in two phases and reached 85.3% intercoder agreement.
The coding produced 42 codes and eight themes from 29 codes about reasons to replicate. The most frequent coded motivation was validating or generalizing results, with 26 mentions. A category covering efficiency or available resources and expected changes over time appeared in 19 mentions.
Eight of the 25 participants said all studies, rather than only a subset, should be replicated. Because participation was incomplete, these responses do not represent all authors of the papers.
The case for clearer rules
The authors say the pattern points to a need for clearer venue guidance and explicit reporting templates. They recommend that replication papers explain why they differ from the original, compare their results with it and contain enough detail to stand on their own.
The review’s reach is limited by its 13-venue search, its chosen terminology and the rule that any change in one study aspect counted as a difference. Missing details in some papers also made classifications difficult. The findings therefore show how replication is being described in this selected literature, not whether exact replications never occur elsewhere or whether clearer guidance would improve practice.
Paper data and sources
Original title: A Meta-Study on Replication Papers in Usable Security & Privacy
Authors: Christian Mack, Benjamin Berens, Hanna Algedri et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text