In a Netherlands laboratory study, participants following generic smart-home security advice often reached an account setting or a companion-app function instead of the device-level or firmware endpoint the advice concerned, according to an arXiv preprint. Participants also frequently judged that they had applied the advice even when their path had stopped at one of those other endpoints.
The researchers tested two advice items. In 84 password sessions, 33 reached no password setting, 50 reached an account-level setting and one reached a device-level setting. In 84 update sessions, 27 reached no update, 19 reached an update in a companion app and 38 reached a verified firmware update.
When the visible action looked like success
The gap between a pathway and a conclusion was clear in the participants’ own judgments. No participant concluded that the password advice did not apply. They judged that they had applied the password advice in 51 of 84 sessions and the update advice in 54 of 84. Of the sessions they judged successful, 49 password sessions ended at account-level settings and 19 update sessions ended at companion-app updates. One applied password session reached a device-level setting, and one reached no password setting.
The password instruction also met a problem of fit with the products themselves. None of the sampled products had a manufacturer-set credential shared across units in the way described by the advice. The sole device-level credential found was unique to that unit. The audit recorded what researchers could locate through the available interfaces and sources, so failure to find another facility did not prove that one did not exist.
Device-to-device differences were just as stark. On one device, 13 of 14 password sessions reached no password setting. On another, 11 of 13 update sessions reached a verified firmware update. These figures are descriptive comparisons across products, not a causal test.
What the test involved
The study involved 28 participants. Each participant worked with three of six devices and both advice items, producing 168 laboratory sessions. The advice came from current national campaigns. The six devices were retrieved from Amazon’s official Best Sellers list and chosen for market popularity rather than confirmed feature availability. Device order and the starting advice were randomized, with a 50 percent chance for each device to start with either advice item.
Participants demonstrated the pathways rather than committing credential changes or installing firmware updates. That kept device states consistent across sessions. Researchers combined observations, think-aloud protocols, interviews and workload assessments. For the numerical analysis, they used descriptive statistics and exploratory regression models; the six NASA-TLX dimensions were treated as separate ordinal measures rather than one combined score.
Support materials were consulted in 66 of 168 sessions. Of 40 password consultations, one returned a device-level password path. Of 26 update consultations, nine returned a firmware path.
Where the evidence stops
The study is best read as a test of what people could determine under controlled conditions, not as a count of completed security changes in ordinary homes. Because the protocol stopped at demonstrations, it did not measure whether participants would later change a password or install firmware. Nor do six devices selected for popularity show how the wider consumer IoT market is organized.
The participant pool was small and skewed toward higher education and prior IoT experience. Twenty-two of the 28 participants held higher-education credentials, and 20 reported prior IoT experience. Those characteristics, along with the urban Netherlands setting, limit how confidently the pattern can be extended to other populations and settings.
One practical lesson is that an account change or app update does not, by itself, identify which part of a product has changed. The research question was whether participants could tell whether advice applied, identify its intended credential or update target, reach that target and establish the resulting device state. In this test, reaching a nearby setting and believing the task complete were not the same as reaching the device-level or firmware endpoint.
A preprint with study materials reported online
The paper is an arXiv version 1 preprint dated 25 August 2026. The authors reported that study artifacts are available through OSF, but video recordings and transcripts will not be deposited. Funding and conflicts of interest were not reported.
Paper data and sources
Original title: "Am I Just That Dumb?": Applicability, Action and Verification in Consumer IoT Security Advice
Authors: Veerle van Harten, Carlos Hernández Gañán, Michel van Eeten, Simon Parkin
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text