A preprint study of an open-source web application found that Cypress end-to-end tests completed far faster than manual execution in the researchers’ local setup, while a locator strategy built around data-cy recorded no repair time for the interface changes tested. The same evaluation also found a small but visible reliability problem: some scenarios did not behave consistently from run to run.
The results come from a case study of Conduit RealWorld, a medium-sized social blogging platform, and a 27-case test suite covering six main feature areas. The work is labeled a preprint and identifies the 9th International Conference on Research in Engineering and Technology (RET 2026).
The speed gap was substantial
Across 20 consecutive automated runs, the full set of 27 end-to-end scenarios took between 1 minute 20 seconds and 1 minute 39 seconds to complete. The average was 1 minute 28 seconds, or 88 seconds.
Manual execution of the same scenarios averaged 6 minutes 8 seconds, or 368 seconds, across three runs. Each of those manual runs was reported as having a 100% passing rate. Comparing the two reported averages, the authors calculated an execution-time reduction of approximately 76.08% for the Cypress runs.
That comparison is best read as a measurement from one local setup, rather than a universal speed ranking. The protocol repeated the automated suite 20 times under the same condition and the manual version three times, resetting the application state to a known baseline before each execution. The study reports averages and percentages, but it does not report inferential tests or confidence intervals.
Most scenarios held steady, with one important exception
The reported flakiness frequency was approximately 7.4%, meaning that a minority of test cases were described as unstable. Over 20 consecutive cycles, 92.6% of scenarios remained stable.
The fluctuating results were concentrated in the Favorite Article function. Test cases #17 and #18 alternated between passing and failing without a pattern, according to the reported results. The evaluation did not determine whether timing, shared application state, backend behavior, network latency or a limitation of the testing framework was the main contributor.
That distinction matters for teams interpreting the findings. The study supports a description of how this suite behaved under the tested conditions; it does not establish that Cypress itself caused the failures or that the same stability profile would appear in other applications.
Locator choice shaped the maintenance result
For the maintainability comparison, the researchers kept the testing logic the same but changed how the suite located elements in the application. Suite 1 used text, HTML attributes, CSS classes and DOM structure. Suite 2 used a data-cy locator, a dedicated marker used to find an interface element in a test.
For the listed interface modifications, Suite 1 took 129.5 seconds, 126.78 seconds, 132.6 seconds, 154.7 seconds and 159.4 seconds to repair. Suite 2 recorded 0 seconds for each listed modification. The reported comparison therefore favored the data-cy suite for the changes examined.
The repair-time result does not by itself show that data-cy locators reduce maintenance costs across software projects. The authors note that repair assessments can be influenced by tester experience, and the comparison was carried out in one application.
A useful result with a narrow reach
The experiment used a local Windows setup with an Intel Core i7 processor and 8GB of RAM. It ran Node.js v22.14.0, Electron 138 and Cypress 15.8.2 with headless browser execution. Those details define the conditions behind the reported timings; they do not describe every environment in which the tools might be used.
The authors’ interpretation is that the speed difference could make frequent or per-commit test runs feasible, while Cypress appeared stable for most of the application’s functions. They connect the fluctuating Favorite Article results with possible timing, shared-state or framework issues, and argue that resilient data-cy locators used with a Page Object Model can reduce long-term repair effort.
The evidence remains limited to one application, one local Windows environment, repeated executions and the listed interface changes. The analysis does not show improved production quality, lower defect rates, greater developer productivity or better user outcomes, and it does not compare Cypress with Selenium or Playwright. Replication across applications and controlled analysis of backend state, network delay and execution order would be needed to clarify how broadly the findings apply.
The paper states that all data are recorded in a Google Sheet, but the supplied report does not list a public data link. No funding statement is reported in the supplied text or metadata.
Paper data and sources
Original title: Evaluating Automated Testing on an Open-Source Web Application Using Cypress
Authors: Quoc-Binh Nguyen, Truc-Ly Phan Nguyen, Ngoc Hong Tran, Dung Hai Dinh
Journal/Repository: 9th International Conference on Research in Engineering and Technology (RET 2026)
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text