Rankings of tested AI fact-checking systems varied across four benchmarks in a preprint comparison, with no single system dominating every dataset. The evaluation brought together sparse baselines, fine-tuned transformers, zero-shot language models and top systems from the AVeriTeC shared task.
An analysis of the tested language models found higher accuracy with gold-annotated evidence than with TF-IDF-retrieved evidence across all reported datasets. The reported differences ranged from 14 to 22 percentage points.
Four benchmarks, different evidence worlds
The four benchmarks represented different evidence settings. AVeriTeC contained 5,783 open-web claims and an evidence corpus of 32,818 web documents; SciFact contained 1,409 claims and 5,183 S2ORC abstracts; ClimateCheck contained 3,199 claims and 394,269 scientific abstracts; and ClimateFEVER contained 7,675 claims and 5,240 Wikipedia pages.
The evaluation separated document retrieval from veracity prediction. Retrieval was scored at cutoffs of five, 10 and 20 documents using gold-document coverage metrics. Veracity was scored with accuracy and macro-F1, an average that gives each label equal weight and was included to address label imbalance. The AVeriTeC protocol also used Hungarian METEOR with a threshold of 0.25.
The evaluation used fixed splits of 80 per cent for training, 10 per cent for development and 10 per cent for testing. Baseline comparisons used example-level 95 per cent bootstrap confidence intervals based on 1,000 samples, while multi-seed experiments reported means and standard deviations.
Retrieval rankings reversed by domain
Retrieval performance differed sharply by benchmark. At the five-document cutoff, the top-five recall score, or R@5, was 0.731 for AIC CTU on SciFact, versus 0.217 for TF-IDF. On AVeriTeC, TF-IDF led with 0.126, while SANCTUARY scored 0.077.
Retrieval was low overall on the two climate datasets, and TF-IDF consistently exceeded BM25 in the reported comparisons. The evaluation therefore reported no universal retrieval winner: the leading configuration differed by benchmark.
Veracity scores moved too
Veracity results also varied by dataset. On SciFact, SANCTUARY led with 0.702 accuracy and 0.700 macro-F1. AIC CTU scored 0.566 and 0.482 on those measures, while fine-tuned models were below 0.47 accuracy and were not significantly better than TF-IDF paired with logistic regression.
ClimateCheck produced a different result. Fine-tuned Longformer and DistilRoBERTa had the highest reported accuracies among the cited configurations, at 0.618 and 0.617. Adding BM25-retrieved evidence was associated with lower accuracy for the tested language models on that benchmark.
Across the reported datasets, SANCTUARY’s macro-F1 was 0.700 on SciFact, 0.482 on AVeriTeC and 0.315 on ClimateCheck. In the cited comparison, SANCTUARY scored 0.709 accuracy against 0.606 for AIC CTU, and 0.482 macro-F1 against 0.371. AIC CTU and SANCTUARY changed order depending on the metric, and no single system dominated across datasets.
Evidence conditions pointed in different directions
Results differed by evidence condition in a matched ablation that held the verifier and prompt constant. Compared with claim-only input, retrieved evidence was associated with a 21.6-point increase for the 8B Llama model and a 6.8-point increase for the 70B model on AVeriTeC. On ClimateCheck, the corresponding changes were declines of 24.4 and 19.4 points.
Another analysis compared words in each claim with words in its evidence using Jaccard similarity, a simple word-overlap measure. Mean overlap was 0.047 on ClimateCheck, 0.122 on ClimateFEVER, 0.087 on SciFact and 0.093 on AVeriTeC. The analysis treats this overlap as a correlate of retrieval difficulty, not a direct measure of veracity.
Where the comparison stops
These figures do not amount to a randomized test of one system or retrieval method being superior. The gold-evidence comparison was against TF-IDF-retrieved evidence, and the matched ablation used fixed samples rather than randomized claim assignment; the differences are benchmark comparisons, not randomized causal estimates.
The scope was also narrow. The analysis covered four benchmarks, and the authors say this limits generalization; large-scale, multilingual or multidomain settings and newer generative settings were not evaluated.
Uncertainty also appeared in the labels. In 40 sampled failure cases, annotator agreement was moderate, with Cohen’s kappa of 0.55 and raw agreement of 0.72; 38 per cent were classified as annotation mistakes or debatable. The sample cannot estimate annotation noise across an entire dataset.
The authors state that they released the code, pre-processed datasets and all results to support reproducible automated fact-checking research. The document is an arXiv preprint, version 1, dated 26 August 2026.
Paper data and sources
Original title: How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation
Authors: Aida Usmanova, Zangir Iklassov, Markus Leippold, Ricardo Usbeck
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text