An internal-state probing method ranked first in a test of AI systems judging whether research ideas were novel, according to an arXiv preprint. Called Think-Probe-Respond, or TPR, it uses a signal from a model’s reasoning phase to condition the final answer. Across the models evaluated, TPR had the highest performance and beat the best competing approach by an average of 22.30%.
The result addresses a specific weakness in the models’ baseline judgments. In the initial analysis, six leading large language models (LLMs) produced poor novelty judgments: the best macro-F1, a score used to compare performance across the novelty classes, was 17.1, and reported mean absolute error values were around one. Predictions clustered in middle classes 3 and 4 on the five-point scale, while the extreme classes, 1 and 5, were rarely identified correctly.
The score did not match the reasoning
The authors describe a gap between what a model represents during reasoning and what it says in its final numerical judgment. The reasoning often contained evidence that could support an accurate novelty assessment, but the final output tended to retreat to safe middle categories. The paper treats this as a mismatch between an internal novelty belief and the score eventually expressed.
A probe between thought and answer
TPR is built around that gap. The model is prompted to reason qualitatively, then a hidden state, an internal model signal from that reasoning phase, is passed to logistic regression, a statistical classifier that predicts the novelty class. The description of that predicted class is added before the model resumes generating its response. The LLM’s parameters remain frozen at inference, and only the classifier is trained, which keeps the added computation relatively light.
The test used RINoBench, a collection of 1,381 expert-authored machine-learning ideas paired with related works, human novelty scores from 1 to 5 and expert-written justifications. The TPR experiments used open-source reasoning and non-reasoning models from several families, including Qwen3, GPT-OSS-20B, Gemma 3 and Llama 3.1.
A clear lead, but no simple size rule
On the benchmark’s main measure, TPR surpassed FineTune and the other tested approaches; its average advantage over the strongest competing approach was reported as 22.30%. The paper reports no confidence intervals or formal significance tests. The result is therefore a benchmark comparison and association, not a formal causal test of whether TPR itself produces better judgments.
Model size was not a reliable guide to the result. Llama-3.1-70B ranked second-worst among the listed systems, while Gemma3-4B ranked second-best. TPR worked with both reasoning and non-reasoning models, but reasoning models had slightly lower average TPR performance. The largest gains appeared for non-reasoning models given explicit step-by-step instructions.
Nor did Chain-of-Thought prompting match the probe. Across the evaluated LLMs, CoT substantially lagged TPR; it offered only marginal benefit for non-reasoning models and had inconsistent effects for reasoning models. Overall, CoT performed below both FineTune and TPR.
The useful signal appeared late
TPR also produced a different spread of predictions. Compared with prompting, it covered the novelty classes more evenly and enabled recognition of very low and very high novelty, although Qwen3 still rarely assigned class 1. The strongest probing point was generally late in the model’s thinking: the final think token was strongest or nearly strongest across nearly all models, while earlier thinking-token probes usually performed substantially worse.
Probes taken during response generation remained competitive but were generally weaker than the final think-token probe. The authors say this pattern is consistent with dilution by surface realization and linguistic planning during response generation.
The gains extended to the accompanying explanations. TPR-generated justifications were broadly aligned with human justifications. Reasoning-capable models generally produced more accurate, elaborate and reliable explanations than smaller non-reasoning models, although performance depended strongly on model choice.
A narrow test, for now
The result remains narrow in scope. The benchmark focuses on machine-learning ideas, so the authors limit generalization beyond that field. TPR requires access to model hidden states, limiting direct use with closed-source LLMs, while the expert novelty annotations are subjective or may be incomplete.
The document is presented as an arXiv preprint, and the paper frames the result as a comparative benchmark association rather than broad validation of scientific novelty judgments.
Paper data and sources
Original title: Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
Authors: Tim Schopf, Tobias Schreieder, Akiko Aizawa
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text