A traffic-video AI system has produced a sharp split across benchmark tests. TAU-Agent placed second on AI City Challenge Track 3, ranked 12th on the out-of-domain FETV benchmark and finished fifth on the out-of-domain PSI-VQA benchmark. The results give a mixed picture: the system reached a high position on one leaderboard, but its standing changed substantially when the benchmark and video setting changed.
A search-and-select pipeline for video questions
At the center of TAU-Agent is a retrieval agent, a component that finds evidence relevant to a question. It coordinates a video-captioning tool and an open-vocabulary tracking tool to retrieve and select that evidence. The main agent then reasons jointly over the question and the retrieved material, refining the frame range, selecting caption segments and object tracks, and scoring each chosen item.
At the final question-answering stage, the model combines frames sampled at different rates with the five highest-scoring caption segments, the five highest-scoring object tracks and optional context from other questions.
Detections from both branches are linked with ByteTrack into object tracks. Each formatted track is sampled at one frame per second and capped at 20 observations. Those tracks then feed the evidence-selection and question-answering stages.
The training data combined AI City Challenge Track 3 and PSI-VQA into one unified training set. FETV was evaluated as an out-of-domain benchmark, while PSI-VQA was also described as out-of-domain in the reported results.
A strong showing on Track 3
On AI City Challenge Track 3, TAU-Agent scored 0.6779 and ranked second. Within that submission, the listed subtest scores varied widely: 1.0000 for BCQ, 0.9500 for MCQ, 0.6685 for BCQ OE and 0.9176 for MCQ OE. It scored 0.5150 on Open QA, 0.5503 on Causal, 0.4314 on Scene, 0.5164 on Temporal and 0.5516 on Summary.
The transfer results are less even
FETV was the clearest weak spot in the reported rankings. TAU-Agent ranked 12th with an overall score of 0.3998; its description score was 0.3513 and its categorical mean score was 0.4484. The paper reports a performance gap relative to the highest-ranked methods, particularly for structured attribute prediction, alongside limited task-specific adaptation to fisheye imagery and structured JSON prediction. The result underscores that performance did not carry over evenly to this benchmark.
PSI-VQA offered a different out-of-domain setting: 40 egocentric dashcam videos focused on pedestrian-crossing scenarios. Its tasks covered crossing intent, cue articulation, relevant-cue choice and decision-critical temporal intervals. TAU-Agent ranked fifth with a final score of 67.9275. Component scores included BCQ F1 of 0.6167, BCQ accuracy of 0.7091, Open QA F1 of 0.7791, MCQ accuracy of 0.7253 and temporal mIoU of 0.5960, a measure of overlap with the relevant time interval.
In one PSI-VQA comparison, the Open QA Cue-F1 score was 0.7791, the highest among listed submissions and 0.1117 above the second-best result. That was the standout subscore in the reported PSI-VQA results, but the overall ranking remained fifth.
The authors interpret the results as evidence of in-domain performance and generalization across traffic-video domains and task formulations. A more measured reading is that TAU-Agent was competitive in selected settings, while FETV remained below the highest-ranked methods. The paper identifies streaming and real-time video understanding as future work; its FETV discussion also highlights limited adaptation to fisheye imagery and structured JSON prediction.
The document is an arXiv version 1 manuscript dated 26 August 2026. The paper states that its code is available at https://github.com/siri-rouser/TAU-Agent.
Paper data and sources
Original title: TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
Authors: Yuqiang Lin, Yan Shi, Sam Lockyer et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text