An artificial-intelligence agent judged not only on its final answer but also on whether its working memory still contained enough information to answer scored highest among the listed Qwen3-VL-4B variants in a new preprint. It reached 53.9% on MMLongBench-Doc and 60.1% on LongDocURL, two benchmarks used for long-document visual question answering.
The proposed signal targets a failure that a final answer can hide: an agent may answer correctly while leaving terminal working memory too incomplete for a later answer. To measure that gap, a frozen reader receives the question and the terminal memory alone, without the trajectory or page images, and its answer is scored against the gold answer.
When a correct answer is not enough
The evaluation covered 1,082 MMLongBench-Doc examples and 2,325 LongDocURL examples. Post-training used a disjoint rendered subset of 2,000 questions, producing 994 supervised-fine-tuning trajectories and 226 retained group-relative policy-optimization groups from a 1,006-question pool unseen during reinforcement learning. The agent used append-only memory updates, could take at most 15 steps and retrieved the top three pages with Jina v4 and Qdrant. Comparators included direct input, a vision-language model with Top-3 retrieval and no memory, supervised fine-tuning and answer-only GRPO.
On a fixed 500-example answerable MMLongBench-Doc subset, the researchers compared full multi-turn runs with runs given gold evidence pages, a setup intended to remove retrieval as a confounding factor. Final-answer accuracy was 33.4% in the full setting and 36.2% with the pages supplied; memory-only accuracy was 27.4% and 30.2%. Pmmc, the memory-missing-correct rate among final-answer-correct cases, was 25.0% in the full setting, or 43 of 172 cases, versus 42.5% with gold pages, or 77 of 181 cases.
A diagnostic built around the last memory
A separate controlled comparison held gold-page access, the answer generator, the reader and the judge fixed while substituting different terminal-memory content. Original agent memory produced final-answer and memory-only positive rates of 36.2% and 30.2%. GPT-4o-improved memory produced rates of 44.4% and 44.8%, while empty memory was near zero on the two readouts. For final-answer rates, the reported 95% Wilson intervals were 32.1% to 40.5% for original memory and 40.1% to 48.8% for improved memory; empty memory was 2.8%, with an interval of 1.7% to 4.6%.
The comparison shows the diagnostic value of separating the answer from the memory that remains available afterward, but it does not by itself establish that the training method caused the difference. The improved-memory condition used a stronger memory-writing model and gold evidence pages, making it a controlled intervention on memory content rather than a direct test of all parts of the trained agent.
Putting memory into the training signal
AWM-GRPO assigns four rewards to the possible combinations of final-answer correctness and memory-only answerability. A correct answer with answerable memory receives 2 reward units; a correct answer with imperfect memory receives 0; a wrong answer with answerable memory receives -0.1; and a complete failure receives -1. Because the reward for a correct answer remains higher than the reward for an incorrect one, final-answer correctness keeps priority while the two correct-answer cases are distinguished. GRPO then normalizes each trajectory relative to the mean and standard deviation of its group, using an epsilon of 10^-6 to stabilize groups with no reward variation.
A controlled synthetic simulation sampled 15,000 groups of eight trajectories for each mixture. In the answer-only version, trajectories were grouped only by final-answer correctness; AWM assigned different advantages to the two cases in which the final answer was correct, depending on whether the memory was answerable.
Higher scores, with a modest memory-quality shift
Across 1,082 full multi-turn MMLongBench-Doc trajectories, supervised fine-tuning recorded 40.8% final-answer accuracy, 38.5% memory-only accuracy and 17.7% Pmmc. Answer-only GRPO recorded 43.2%, 42.5% and 19.9%. AWM-GRPO recorded 45.4%, 44.5% and 17.2%, respectively. In this analysis, the AWM-trained variant therefore had both higher answer rates and a lower share of correct answers unsupported by its terminal memory.
The same comparison under controlled gold-page access used 733 answerable examples. Answer-only GRPO reached 45.4% final accuracy, 41.2% memory-only accuracy and 19.1% Pmmc; AWM-GRPO reached 48.0%, 43.5% and 16.4%. Reported training dynamics also showed memory-only accuracy rising from 38.8 at step 40 to 43.5 at step 280.
The reported AWM-GRPO margin over direct input on MMLongBench-Doc was 10.4 percentage points, compared with 8.1 points over Top-3 retrieval, 4.7 over supervised fine-tuning and 2.3 over answer-only GRPO. On LongDocURL, the margins were 11.9 points over retrieval, 4.4 over supervised fine-tuning and 2.7 over answer-only GRPO. These are reported score differences within the benchmark comparisons, not causal estimates.
What the test does not establish
The document is arXiv preprint arXiv:2608.25618v1, dated 26 August 2026. Its memory audit found that all 500 improved memories cited an eligible gold-evidence page, but it did not test whether the claims in those memories were actually entailed by the page or whether the memories were complete. After an exact-answer-string filter, 98 of 304 cases scored positively, or 32.2%; the paper notes that this was not a like-for-like estimate for the full set.
That distinction matters because memory-only answerability is not the same as claim-level source grounding. The report also lists different final-answer accuracies for apparently corresponding MMLongBench-Doc variants: 53.9% for AWM-GRPO in one reported comparison versus 45.4% in the memory-quality analysis, and 51.6% versus 43.2% for answer-only GRPO. The discrepancy remains to be reconciled.
The results come from computational evaluations using publicly released benchmarks; the paper says it collected no new data, involved no human subjects and did not target deployment. The reported evidence concerns Qwen3-VL-4B variants on the two named benchmarks, so it does not establish human performance or deployment benefit. The paper states that its code is available at github.com/DongzhuoranZhou/AWM.
Paper data and sources
Original title: AWM: Answerable Working Memory for Long-Document VQA Agents
Authors: Dongzhuoran Zhou, Yuqicheng Zhu, Yule Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text