Large language models were generally better at finding basic study information than statistical results in a benchmark of meta-analysis data extraction. The authors’ proposed workflow puts study information in Tier 1, with minimal oversight, but places statistical results in Tier 3, requiring prompt tuning and manual validation.
Recall—roughly, the share of relevant data fields a system found—ranged from 0.52 to 0.84 for study information, 0.70 to 0.78 for quality assessment, and 0.21 to 0.76 for statistical results. Customized EXT, the setup using domain-specific prompts, was reported as the strongest approach across all three categories.
A benchmark built from trial evidence
The benchmark drew on six meta-analyses published between 2021 and 2025. They covered three medical areas, with two meta-analyses from each area, and yielded 58 original randomized controlled-trial manuscripts.
To create the reference data, the researchers manually extracted information from all 58 trials, had two trained research staff check it independently, and validated it with medical experts. The dataset was built from the trial reports rather than copied from the published meta-analyses.
Three LLMs were tested with four setups: basic prompting, self-reflection, a combined approach using outputs from different models, and customized prompts. The systems were judged field by field against the reference data in three categories: statistical results, quality assessment, and basic study information.
Tailored prompts led the comparison
Against the baseline EXT setup, the reported average changes were +14.8% in recall and −0.8% in precision for Customized EXT. Combined EXT recorded +5.9% in recall and +2.0% in precision, while self-reflection recorded +1.8% and +0.1%, respectively. Recall describes how much relevant information was found; precision describes how often the extracted entries were correct.
An overall comparison found that recall differed across the four methods (reported χ²(3)=9.81, p=0.0203). A follow-up test identified a significant difference between the baseline EXT and Customized EXT, but not between EXT and Combined EXT.
Most errors were omissions
Across the models, Grok ranked best overall, with a mean recall rank of 1.3, followed by Gemini at 2.0 and GPT at 2.7.
The clearest warning came from the error breakdown. Missing fields accounted for 87.8% of analyzed error cases, or 19,470 instances. Incorrect values made up 10.3%, or 2,296 instances, while overgeneralized errors accounted for 1.2%, or 267 instances, and incorrect units for 0.7%, or 163 instances.
Human review stays in the loop
The researchers also checked the automated evaluation against human review. In a blinded sample of 900 fields, LLM-assigned labels agreed with human reviewers 96.09% of the time; agreement between the two human reviewers was very high, at Cohen’s κ=0.987.
The proposed workflow assigns study information to Tier 1, with minimal oversight; quality assessment to Tier 2, with human review; and statistical results to Tier 3, requiring prompt tuning and manual validation.
What the benchmark does not answer
The benchmark’s reach is limited: its recall ranges and prompt changes came without confidence intervals. The authors flag limited domain coverage, the challenge of information embedded in charts and figures, the use of black-box models without fine-tuning or intermediate outputs, and the possibility that newer models could perform differently.
The study did not systematically assess computational costs, API expenses or processing times. With those questions still open, the proposed tiers keep human review in the loop, with the greatest scrutiny reserved for statistical results.
Paper data and sources
Original title: What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction
Authors: Li L, Mathrani A, Susnjak T
Journal/Repository: Research Synthesis Methods
Status: Peer-reviewed
First online: 2026-08-19
DOI: Not available
Original paper · Full text