Broad financial capability scores do not fully determine how models perform on the specific jobs involved in risk and compliance review, a new preprint reports. The study asks whether operation-specific and evidence-state tests reveal model-selection information that broad financial scores miss.
FinRiskAtlas contained 9,742 static instances in 53 task families. Forty-two families covered Domain Knowledge, while 11 covered downstream operations. Those downstream tasks were defined around concrete review outputs under specified evidence conditions.
One overall score does not settle the choice
The downstream-operation rankings were only partly aligned. Their mean pairwise Spearman correlation was 0.42, a measure of how similarly two rankings line up, and 37 of the 55 operation pairs had correlations below 0.5. A model that ranks well for one review job may therefore rank differently for another.
The study also measured the regret from using broad Domain Knowledge scores to shortlist models for a particular operation. When only one model was shortlisted, the observed gap between that choice and the best available configuration was 18.01 points for information extraction, 11.21 points for quantitative reasoning and 8.47 points for legal-outcome prediction. Decision-view generation still showed a 6.70-point gap when the shortlist contained 15 models.
Each downstream family was specified before evaluation through a contract covering the evidence visible to the model, the professional decision object, the reviewer artifact and the scoring method. The experiments covered 33 model configurations, using direct zero-shot answers without in-context examples or requested chain-of-thought. Each model-instance pair was evaluated once.
The harder question may be when to ask for evidence
FinRisk-Ask adds another test: whether a model asks for more evidence or proceeds with the material already available. The released benchmark contained 680 states from 104 trajectories, including 583 recorded Ask states and 97 recorded Proceed states.
Future evidence was withheld during inference and used only to construct expert-verified evidence targets. The offline replay can therefore compare a model's choice and evidence request with a target derived from the later trajectory, but it does not measure what happens after a request is carried out.
That distinction mattered. Among configuration pairs whose balanced recorded-action agreement, or BAcc, differed by no more than 0.1 points, evidence-request alignment, or ERA, differed by as much as 28.31 points. BAcc captures agreement with the recorded action, while ERA focuses on whether an evidence request aligns with the relevant need.
The model-level results show the trade-off. Ling-2.6-1T recorded an ERA of 79.75%, with Ask recall of 96.57% and conditional request alignment of 82.58%. Qwen3.7-Max had the highest conditional request alignment, at 87.67%, but entered the recorded Ask branch on only 57.80% of Ask states, leaving its ERA at 50.68%.
Models were also uneven in how well they recognized the two branches. Across 32 non-evaluator configurations, the median difference between Ask recall and Proceed recall was 46.26 points, and 24 configurations had gaps larger than 20 points. The study presents this as descriptive aggregate behavior across the tested configurations.
What the release leaves open
Six domain experts contributed to the benchmark's taxonomy and quality control, and every retained instance was independently reviewed by at least two relevant experts.
The offline design sets a clear boundary around the evidence. Future material was withheld while the model made its decision and was used only afterward to construct evidence targets, so the evaluation does not establish the effect of obtaining new evidence in a live review.
Raw enterprise trajectories were not released. The package provides only de-identified reconstructed states and evaluation artifacts, leaving the original trajectories unavailable for direct inspection.
The document is identified as arXiv:2608.25325v1 and dated 26 August 2026. Funding and conflict-of-interest statements are not reported in the supplied document.
Paper data and sources
Original title: FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review
Authors: Suyang Zhong, Jingzhe Zhu, Qi Xu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text