A new test of AI theorem-proving systems points to a sharp divide between answering a mathematical question and producing a formal proof that passes Lean 4 checks. On the MathAdv benchmark, the highest reported Lean 4 accuracy was 21.88% for Goedel-Prover-V2. DeepSeek-Prover-V1.5 RL plus RMaxTS reached 16.56%.
MathAdv was built to separate several abilities that can look similar from the outside: mathematical knowledge, informal reasoning, formal proof construction, and the ability to handle mathematically equivalent reformulations. Its results suggest that success in ordinary-language problem solving does not automatically carry over to formal proof construction.
A benchmark with several ways to test the same skill
The benchmark contains 321 problems across 13 mathematical domains. Of those, 298 were formalized in Lean 4, while 23 were deferred for auxiliary evaluation because of gaps in current Mathlib support, the formal mathematics library used in the benchmark.
Suitable problems could receive up to three additional tests: fill-in-the-blank reasoning, multiple-choice questions aimed at mathematical knowledge, and expert-crafted transformed problems. The multiple-choice set covered 293 problems, while the transformed set covered 30. These transformed versions tested whether a model would remain successful when the same mathematics was presented in an equivalent form.
The formalization process used a four-stage, language-model-assisted pipeline with a verifier in the loop. It included compiler correction, independent semantic checks, and a final expert review, turning the source problems into formal Lean statements.
The evaluation included models that generate individual proof steps, models that attempt whole proofs, and general-purpose language models. Whole-proof results were summarized with pass@K, a metric based on making several attempts.
The formal test was much harder than answering in prose
The contrast was especially clear for the general-purpose systems tested on auxiliary tasks. GPT-5.4 recorded 64.2% on direct answers and 82.88% on multiple-choice questions. DeepSeek-R1 recorded 58.0% and 51.71%, while DeepSeek-V3.2 recorded 66.7% and 75.00%. Those figures were substantially above the reported Lean accuracies, although the supplied results do not state the direct-answer denominator or report confidence intervals.
Performance also varied sharply by subject. Goedel-Prover-V2 was strongest in number theory and linear algebra, but every model scored 0% on topology. The benchmark therefore did not produce one uniform picture of mathematical ability: performance depended heavily on the domain being tested.
Within the DeepSeek-Prover-V1.5 line, the reported accuracy was 1.25% for the base model, 11.25% for its reinforcement-learning variant, and 16.56% with RMaxTS. Goedel-Prover-V2 had the highest reported score at 21.88%. The study presents these as benchmark comparisons, without confidence intervals or significance tests.
Equivalent wording exposed another weakness
Changing the wording and structure of a problem could also unsettle a model. Goedel-Prover-V2 solved the original but failed the transformed version in six reported cases, with no cases in the reverse direction. For DeepSeek-R1, the corresponding counts were four and zero. The paper notes that the absolute counts are small, and the transformed set contained only 30 problems.
The benchmark also compared proof generation with and without interactive Lean feedback under the reported equal-budget setup. DeepSeek-V3.2 rose from 5.00% without interaction to 7.50% with it, while GPT-5.4 rose from 9.06% to 13.75%. The comparison is limited by the lack of confidence intervals or statistical tests and by the fact that the interactive setting differs from the main benchmark settings.
Natural-language hints did not have a single consistent effect. Performance was higher for the two DeepSeek models when hints were used, but lower for Goedel-Prover-SFT and Goedel-Prover-DPO. The supplied results do not include the numerical values for this experiment.
What successful and failed proofs looked like
The qualitative error analysis identified four recurring failure patterns: proofs that stopped before they were complete, tactic or command choices that did not fit the problem, invented declarations for library results, and repeated commands that made no progress.
When proofs worked, they were described as relying on exact results from the formal library, explicit step-by-step arguments, or simplification after formal definitions had been unfolded. A separate junk-theorem probe tested compilation prediction and semantic understanding across 360 calls to four models. Compilation-prediction accuracy was 58% for gpt-5.5 and 4% for each listed DeepSeek model; deepseek-v4-pro scored 1.91 out of 2.00 for semantic understanding.
The semantic-understanding score in that probe was hand-graded on a 0-to-2 scale, and the supplied analysis does not report inter-rater reliability. It is best read as a probe-specific result, not a standalone measure of broad mathematical understanding.
A useful diagnostic, with a narrow reach
The authors identify two main limitations. MathAdv is modest in size because expert creation and validation are costly, and current Mathlib coverage prevents some advanced problems from being formalized. That makes the benchmark a bounded test of the sampled tasks rather than a complete picture of theorem proving across mathematics.
The document is an arXiv version 1 preprint dated 26 Aug 2026. The paper reports that both the dataset and the evaluation scripts are available through the MathAdv GitHub repository.
Paper data and sources
Original title: MathAdv: What Theorem Provers Know, Reason, Formalize, and Generalize
Authors: Jiaxin Yuan, Connor Martinez Lockhart, Xiaoyu Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text