Seven vision-language models struggled to reconcile first-person video with user dialogue when the two sources pointed to different answers, according to a benchmark designed to test AI assistants. Mean understanding accuracy was 63.0% on multimodal-grounded examples but 18.5% on contradictory ones. Every evaluated model scored below random guessing in the contradictory scenario except Molmo2-8B, which reached 24.9% on the four-choice task.
A benchmark built around conflicting signals
The benchmark covered two related tasks: understanding first-person scenes and deciding whether, when and how to help. Its evaluated split contained 7,767 rows, including 6,978 real-video multiple-choice question-answering examples and 789 synthetic VISTA Assist-Step rows. The real-video portion combined EgoPlan-Bench2, QaEgo4D and MIntRec, which contributed 3,000, 3,000 and 978 rows respectively. The decision portion was generated with VISTA and paired with oracle assistance metadata. Seven VLMs were evaluated in both parts.
Sixteen human annotators reviewed the constructed examples, with each example assigned to three annotators. Corrected answers were aggregated by majority vote. The review checked the benchmark answers, but the VISTA episodes remain constructed scenarios rather than records of people receiving assistance. The setup tests whether a model can select the more trustworthy modality, predict a next action and decide whether, when or how to intervene.
Text-grounded questions scored highest
The text-grounded cases had the highest mean accuracy among the reported understanding scenarios. Accuracy was 71.0% for text-grounded examples, compared with 41.5% for video-grounded examples marked off-topic and 45.2% for those marked on-topic. In this benchmark, questions grounded in dialogue were answered more accurately than either video-grounded group.
An oracle diagnostic compared the full input with benchmark conditions in which a distractor modality was removed. Across all four models, the contradictory-dialogue removal condition had accuracy 34.0 to 43.0 percentage points higher than the full-input condition. Differences for off-topic and text-grounded examples stayed within 2.0 points, while the on-topic removal condition was up to 4.5 points lower. Because this was an oracle diagnostic rather than a randomized intervention, it describes a contrast between benchmark conditions and does not establish that removing dialogue caused the difference.
The decision to intervene was uneven
The benchmark also showed wide differences in decisions about assistance. On VISTA, assistance-decision F1 ranged from 0.8117 for Gemini 3.1 Flash Lite to 0.1920 for Qwen3.5 2B. The error analysis described tendencies toward early intervention and difficulty staying silent in benign scenes.
The errors also differed by model. Gemini-3.1-Pro had the fewest errors overall but the highest rate of no-assistance false alarms. Qwen3.5-Plus had low false-alarm rates but concentrated missed interventions, while InternVL3.5-4B had the largest overall errors.
Internal signals did not guarantee the right answer
Researchers also used layer-wise probes to test how well each model's own soft answer preference could be recovered from its internal representations. For Qwen3.5-2B, probe accuracy jumped from 49.5% at layer 15 to 94.5% at layer 16. Three 36-layer models showed a similar transition at layer 25, with probe accuracy ranging from 60.0% to 69.5% before the transition and from 93.5% to 97.0% after it. Peak probe accuracy reached 97.5% to 100.0%.
Those results measured recoverability of a model's preference, not whether that preference was correct. The probes targeted each model's own soft answer preferences rather than gold correctness, and representation separation did not guarantee correct modality arbitration.
The follow-up fixes were mixed
Uniform visual-token attention reweighting was sensitive to ε and did not uniformly improve the scenarios. At ε = 10, the reported accuracies were 36.8% for multimodal-grounded examples, 18.5% for contradictory examples, 26.6% for off-topic video-grounded examples, 27.6% for on-topic video-grounded examples and 38.7% for text-grounded examples.
In a separate NaPO preference-alignment test, overall accuracy was 48.9%, compared with 51.3% at baseline. Text-grounded accuracy rose from 73.7% to 75.4%, but the other scenario accuracies were lower under NaPO. The attention-reweighting and NaPO analyses were specific to Qwen3.5-2B.
What the benchmark does not show
The benchmark's scope is narrow. The understanding task is four-choice video question answering, and the VISTA videos are synthetic. Layer-wise probing covered four models, while the attention and NaPO analyses were specific to Qwen3.5-2B. Constructed contradictory dialogues are diagnostic scenarios rather than faithful transcripts of natural user behavior. The benchmark measures model performance, not human safety, usefulness or reliability in deployment, and it does not cover open-ended assistant behavior.
The document is an arXiv version 1 preprint dated 26 August 2026. The authors state that the dataset is publicly available at https://github.com/NYCU-NLP-Lab/EgoArgus.
Paper data and sources
Original title: EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
Authors: Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text