An artificial-intelligence system designed to learn reusable procedures for scientific hypothesis work recorded the highest reported scores for hypothesis generation and test pass rate in a 13-task computational benchmark. It did not record the highest execution success rate: POPPER scored higher on that measure.
The results come from a version 1 arXiv preprint dated 26 Aug 2026. The evaluation covered computational benchmark tasks, so it does not show that HypoForge discovered valid new scientific knowledge in human research or that the benchmark hypotheses would be confirmed outside those tasks.
A system built around reusable skills
HypoForge is presented as a stage-wise skill-learning framework for automated scientific hypothesis generation and testing. It is intended to distill experience from proposing a hypothesis, designing an experiment and carrying it out into procedures that can be reused.
The framework treats generation and testing as different jobs. For generation, agents use adversarial self-improvement, comparing and challenging proposed hypotheses without explicit supervision; for testing, the system uses empirical feedback from outcomes, including whether an experiment runs and whether its result meets p < 0.05.
The benchmark
The evaluation used all 13 tasks in HypoBench. Generation used representative data from the selected tasks, while testing retained only tasks with empirically evaluable reference hypotheses.
Selected tasks were divided between training and unseen testing sets at a 1:1 ratio. The training set supplied experience for skill learning, and the unseen set was reserved for evaluation.
For generation, the paper compared HypoForge with HypoGeniC and HypotheSAEs. For testing, it compared the system with POPPER.
Generation was scored with Q(Ht), a quality measure, and Hit@K, a measure of how often proposed hypotheses covered the benchmark's reference hypotheses. Testing used Test Pass Rate (Th) and Execution Success Rate (Eh); a test counted as successful only when its experiment ran and the result met p < 0.05.
The strongest reported results
HypoForge's reported generation scores were Q(Ht) = 0.785 and Hit Rate = 0.648, the highest reported values in the evaluation. Its reported coverage advantage over HypoGeniC was 17.6%, and over HypotheSAEs it was 44.0%.
On hypothesis testing, HypoForge had the highest reported test pass rate, Th = 0.659, and an Execution Success Rate of Eh = 0.966. POPPER had higher execution success but a lower test pass rate.
The full generation variant recorded Q(Ht) of 0.785 versus 0.726 without discriminator feedback, while Hit@K was 0.648 versus 0.491.
In the corresponding testing comparison, the full variant had Th = 0.659 versus 0.565 without execution outcomes, while Eh was described as comparable.
Across the reported refinement trajectory, generation Q(Ht) moved from 0.612 to 0.808 and Hit@K from 0.208 to 0.679. The paper notes intermediate fluctuation, and these are trajectory endpoints rather than a controlled between-method estimate.
The testing trajectory likewise ran from Th = 0.395 to 0.669 and Eh = 0.516 to 1.000. These figures are trajectory endpoints, not formal between-method estimates.
What the numbers do and do not establish
The reported averages came from three independent runs. All agents used DeepSeek-V4-Flash; updates were retained only when the learning objective improved, and training stopped after three consecutive attempts without improvement.
The supplied analysis reports no formal between-method significance tests, confidence intervals, preregistration or power calculations. It also says the table's reported plus-or-minus term is undefined. The numerical gaps should therefore be read as reported benchmark differences, not established statistically significant gains.
The study focuses mainly on data-driven discovery with executable computational experiments. It does not establish performance in complex theoretical reasoning, wet-lab research or interdisciplinary knowledge integration.
Independent reproduction and testing on additional benchmarks would be needed to assess how broadly the results apply. The analysis also leaves open how feedback reliability might be improved through stronger evaluators, multi-agent consensus or human input, and how scientific skills should be structured and combined in more complex discovery pipelines.
For now, the evidence is limited to the stated HypoBench tasks, models, data splits, baselines, metrics and three-run averages. It is relevant to the design of self-improving AI agents, but it is not evidence of human or clinical outcomes and does not establish that benchmark hypotheses would be confirmed outside the reported computational tasks.
Paper data and sources
Original title: HypoForge: A Self-Improving Multi-Agent Framework for Automated Hypothesis Generation and Testing via Scientific Skill Learning
Authors: Ziqing Qian, Jiaying Lei, Yifang Wang, Nan Cao
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text