High overall scores on a new finance benchmark masked a steep drop on the hardest questions. Leading models answered more than 97% of Easy items correctly, but the best full-coverage score on the Hard band was 34.68%. Even after the researchers narrowed the comparison to 372 Hard questions with complete local context, the best score was 54.57%.
On all 10,198 questions, Gemini-3.1-Pro led at 85.29%, followed closely by GPT-5.6-Sol at 84.75%. The strongest open-weight reasoning model reached 75.76%, while the leading finance-specialized model scored 68.07%. The spread suggests that a strong average score can conceal where a system breaks down.
The scores split sharply by difficulty
FinExam-10K contains 10,198 expert-reannotated questions aligned with CFA Levels I to III and FRM Parts I to II. Of those, 5,110 were released and 5,088 were held out for the leaderboard. The construction process included rationale filtering, normalization and global deduplication, followed by two-stage review of every retained record by a four-member finance-qualified team.
Researchers evaluated 10 closed-source proprietary models, two open-weight reasoning models and five finance-specialized models. Each answered the same 10,198 Full-Coverage questions. The intervention branches used DeepSeek-R1 CoT and GPT-4o PoT, with deterministic decoding where supported and a fixed answer-extraction cascade.
The benchmark's frozen empirical partition included 6,578 Easy items, 2,183 Medium items and 1,437 Hard items. Checks using different weighting rules found a stable ordering: the adopted rule agreed with the flat 17-model mean on 95.8% of items, with a Spearman rank correlation of 0.998, and with an equal-weight three-family rule on 90.8%, with a correlation of 0.991.
The Hard result has a substantial context caveat. Of the 1,437 Hard records, 1,065 lacked locally attached parent evidence. In a manual audit of 30 items from each difficulty band, context was incomplete in 1 Easy item, 5 Medium items and 17 Hard items, or 3.3%, 16.7% and 56.7%, respectively. That means the full Hard-band score can combine missing material with difficulty in reasoning from the material that is available.
Many hard errors pointed to the same wrong answer
Errors on the hardest complete-context items were not spread evenly across the wrong choices. Among 369 such items with a defined concentration measure, the median concentration of erroneous votes was 0.923, compared with 0.605 under an item-specific null. The observed figure exceeded the null on 346 items, and all erroneous votes landed on one distractor in 148. The one-sided exact sign-test probability was 1.9 times 10 to the minus 75.
The universal-failure core contained 188 questions that all 17 models missed. Of these, 141 were context-incomplete and 47 were context-complete; on 41 questions, every model selected the same wrong option. The pattern documents convergence in the models' answers, but it does not establish a single shared internal mechanism behind those mistakes.
Retrieval was useful in some conditions, but risky overall
Adding retrieval did not deliver a dependable average gain. On Full Coverage, DeepSeek-R1 CoT Function-RAG produced 505 rescues and 538 harms, for a net change of minus 0.32 percentage points (p = .322). FunctionGraph-RAG produced 509 rescues and 500 harms, for plus 0.09 points (p = .801). For GPT-4o PoT, the Function-RAG and FunctionGraph-RAG variants were 1.27 and 1.30 points below Direct, while informed verification reached 68.92% accuracy, 0.45 points below Direct. The comparisons used matched items and two-sided exact McNemar tests on the discordant pairs.
The average hid a more specific pattern. When retrieval retained no function, a subgroup representing 68.7% of items, the three-function PoT graph result was 1.10 points lower in accuracy (p = .0040). When exactly one relevant function was retained, graph expansion was 2.92 points higher (p = .0007). Both comparisons remained significant after Benjamini-Hochberg correction. In other words, the result depended on whether the retrieved material was judged relevant, an association that does not guarantee a gain on a new question.
A selective gate kept the branch use limited
To make the intervention selective, the researchers tuned a Direct-conditioned gate on the 5,110 public items using five-fold out-of-fold performance, then froze it before testing the 5,088 held-out items. The gate invoked the graph branch on 404 held-out questions, or 7.9%. Direct accuracy was 70.83%, compared with 71.23% for the gated result, a net increase of 0.39 points from 55 rescues and 35 harms (p = .0446). On the 4,219 held-out questions with complete context, the trigger rate was 4.3% and the gain was 0.55 points (p = .0003). The reported 1.08-times figure is an implied branch-call count, not a measured cost or latency.
More elaborate checks did not show a reliable end-to-end improvement. In a 369-item verification probe, the system flagged 13.6% and revised 12.7%, producing 11 rescues and 10 harms for a 0.27-point change (p = 1.000). An abstention probe had 85.9% selective accuracy, but falsely flagged 53.3% of answerable items. A plan-then-solve probe instead changed accuracy by minus 3.33 points (p = .353).
The result stays within the benchmark
The supplied document is an arXiv version 1 preprint dated 28 Aug 2026. Its percentages describe this 10,198-question benchmark, its released and held-out partitions and its panel of models. They are benchmark comparisons under those conditions, not population-wide estimates for every future model or direct evidence about real-world financial decisions.
Paper data and sources
Original title: FinExam-10K: When Retrieval Helps Financial Reasoning?
Authors: Yan Lin, Jingyu Sun, Zhongliang Guo et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text