An AI-assisted study of graphical user interfaces (GUIs)—the screens and controls people use in software—reports two separate results. In a retrieval benchmark, trained BERT-LTR models were reported to rank candidate GUIs above BM25 on mean metrics. In a controlled study of 19 participants, the RaWi prototype was associated with statistically significantly larger counts of certain GUI components than a traditional high-fidelity prototyping approach. Those counts were productivity proxies, or stand-ins, rather than direct measures of time saved, implementation cost or productivity in a software team.
The work targets two related tasks: reducing the effort needed to turn natural-language requirements into GUI prototypes, and reducing the effort needed to verify requirements in GUI applications and prototypes. It brings together systems that use language and images, along with a multimodal large language model (MLLM) agent that creates and evaluates interaction trajectories—sequences of actions through an interface.
Ranking candidate interfaces
To build the retrieval test, the researchers collected relevance judgments for 450 natural-language requirement queries, pairing each query with 30 GUIs. Forty-nine workers supplied 40,500 individual annotations. The final gold standard contained 100 queries, each linked to the top 20 GUIs used for evaluation.
The evaluation compared a baseline, adapted PRF-KLD/AQE methods, SBERT and several BERT-LTR variants. It used average precision (AP) and mean reciprocal rank (MRR), among other standard information-retrieval metrics. Model differences were tested with the non-parametric Wilcoxon signed-rank test, with a Holm correction applied separately for each metric across pairwise comparisons.
Across the mean metric values, the trained BERT-LTR models were reported to outperform BM25. The pairwise BERT-LTR (2) variant was reported to differ significantly from BM25 on AP, H@5 and H@10, P@7 and P@10, and NDCG@3, NDCG@5, NDCG@10 and NDCG@15.
On the reported BERT-LTR (2) row, AP was 50.1 and MRR was 63.1. These are benchmark scores for ranking candidate GUIs; they are not evidence that the method has been deployed or improved outcomes for software stakeholders.
A small prototype comparison
Separately, RaWi was evaluated in a controlled within-subjects experiment, meaning the 19 participants compared it with a traditional high-fidelity GUI-prototyping approach. The approach-and-task order conditions—A1B2, A2B1, B1A2 and B2A1—were randomly assigned with an even distribution.
In that comparison, the proposed approach was linked to statistically significantly larger productivity-proxy counts at all considered time steps for component counts (a), (b) and (d), with a p-value below 0.05. Cliff’s delta indicated a large difference. The result concerns component counts rather than direct measures of productivity, elapsed time or implementation cost.
RaWi’s reported System Usability Scale (SUS) score was 81.57, compared with an average SUS of 68. The score was self-reported by participants, so it reflects perceived usability in this test rather than establishing performance across software teams.
Verification and reranking
The wider system also describes a multimodal LLM agent for verifying complex GUI requirements. It does this by creating and evaluating interaction trajectories, or sequences of actions through an interface, to address functional and non-functional requirements. The abstract reports high effectiveness on a human-annotated benchmark, but the supplied extract does not include the detailed accuracy figures or the full later evaluation.
Another method, GUI-ReRank, combines automated multimodal annotation and embeddings with constrained multidimensional retrieval, including negation, before using image and text representations to rerank results. Its benchmark comprised 100 natural-language requirement queries, each paired with 20 Rico GUIs, and enabled a direct comparison with BERT-LTR.
In the GUI-ReRank evaluation, the GPT-4.1 row reported AP 81.3 and MRR 92.7. The surrounding GUI-ReRank results and cost discussion are incomplete in the supplied extract, so those figures alone cannot establish the method’s overall advantage or practicality.
The evidence stops short of production claims
The retrieval component was tested on the Rico dataset. The analysis notes that pooling and filtering can exclude relevant GUIs, while crowdsourced relevance judgments can be affected by worker selection and subjectivity.
RaWi’s component counts are proxy outcomes, and the controlled study was limited to two application domains. The supplied analysis calls for more diverse user evaluations. It does not show improved stakeholder outcomes, project delivery time, software adoption or production software quality.
The supplied text reports that the Chapter 3 interactive prototype, source code, datasets including the retrieval gold standard and a demonstration video are publicly available. Open questions include how well the methods generalize beyond Rico and across more diverse application domains, and how they perform in real requirements work.
Paper data and sources
Original title: From Natural Language Requirements to Graphical User Interfaces: Automated Prototyping and Verification with Pretrained Language Models
Authors: Kristian Kolthoff
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text