A visual-first artificial intelligence system found the right civil engineering plan page in nearly 93% of test cases, far ahead of text-based and hybrid search methods. But the same system was much less dependable when it had to determine which numerical rule governed a real project plan. The contrast is the study’s central result: visual search was strong, while autonomous rule extraction on real plans was not.
The results come from PlanSightRAG, an arXiv version-1 preprint dated 26 August 2026. The system directly indexes plan images and combines visual retrieval, MaxSim heatmap grounding to show relevant image regions, an agentic Planner–Retriever–Auditor–Synthesizer pipeline, and autonomous rule-grounding, meaning it attempts to find the governing rule and threshold itself, in an OCR-free compliance system.
Finding the right drawing was the easy win
On a page-disjoint test split, the system’s ColNomic-3B visual retriever reached 92.69% Recall@5, meaning the correct page appeared among its first five results in that share of cases. That was 55.90 percentage points above the strongest text retriever, BGE-M3 combined with optical character recognition, which scored 36.79%, and 69.58 points above the VisionRAG hybrid baseline at 23.11%. The result cleared the study’s pre-declared 15-point target for retrieval.
The apparent strength was not confined to the agencies represented in the main index. The study assembled a 4,056-pair question-and-answer benchmark across five state departments of transportation, built over a 1,898-page index, and separately tested 93 verified Michigan transfer pairs. Recall@5 was 91.47% in distribution and 91.40% on held-out Michigan data over the combined index, although the Michigan candidate pool was smaller and was not directly comparable to the joint-index result.
The benchmark measured more than search. Answers were scored for binary correctness by a Qwen2.5-VL-72B language-and-vision model acting as a judge. In a 424-pair prompting comparison with retrieval, judging and decoding held fixed, the strongest recipe asked the model to critique and correct itself. It reached 82.31% judge accuracy, 5.42 percentage points above its zero-shot result.
Controlled compliance tests did not tell the whole story
The compliance results were strongest in controlled conditions. With a recipe using a pre-resolved numerical threshold for each generated drawing, verdict accuracy was 100% on a 500-drawing single-document set, a 100-drawing multi-plan set and a 50-drawing near-threshold stress set. A non-visual-model baseline that used OCR and thresholds reached 76.4% on the 500-drawing set, compared with 100% for the visual-model recipe.
A separate agentic test put the system through a small set of valid designs and synthetic failures. It produced the correct verdict on all eight PASS queries and identified 79.17% of 24 mutated FAIL queries. It was more sensitive to changed dimensions or notes, at 87.5%, than to symbol substitutions, at 62.5%.
The study also tested autonomous rule-grounding against a clean 1,913-candidate corpus of specification sheets and plan pages. Rule retrieval was 80.0% at rank one and 100.0% within the first five results; numerical-limit extraction and self-grounded verdict accuracy were also reported at 100%.
Real plans exposed the bottleneck
On a selected set of real WYDOT project plans, the system retrieved a topical standard within its top five results for 10 of 12 items, or 83.3%. When supplied with the threshold, it audited 12 real-plan overlays—six compliant and six violating—with 100% verdict accuracy. But when it had to extract the governing value itself, it found the correct value in only three of nine numerical-limit items, or 33%, and produced correct autonomous verdicts for 44%.
The real-plan result leaves a clear split: topical retrieval and auditing with supplied thresholds were stronger than autonomous extraction of governing values. The weak point was rule selection, not the final comparison when a threshold was already supplied.
More adaptation and higher resolution were not automatic fixes
The authors also tested LoRA, a method for adapting a model to a domain. Across three LoRA configurations trained on 3,211 in-domain pairs, adaptation did not improve the main retrieval result. The full-language-model LoRA configuration instead produced a 37.03-percentage-point in-distribution drop.
Increasing visual detail through tile-level retrieval produced modest gains, not the 20-point improvement set in advance as the study’s H4 target. On the 424-pair test split, judge accuracy rose from 64.23% with full-page images to 68.77% with tiles, while Recall@5 increased from 76.89% to 82.08%. The tiling comparison used the predecessor ColPali backbone.
A promising retrieval layer, not an autonomous reviewer
The evidence is computational and document-based: retrieval, visual question answering, generated compliance drawings, specification grounding and limited real-plan auditing. Much of the benchmark material was machine-generated or automatically verified rather than fully human-annotated. Structured human validation of 4,656 generated items remains ongoing, while the 78-pair WYDOT anchor set is fully human-grounded. The real-plan evaluation was also small and selected.
The study does not show reliable fully autonomous compliance checking on dense real standards, superiority to human engineering review, or that controlled 100% verdict accuracy generalizes to arbitrary rules or real regulatory books. The eight-query PASS result is too small for reliable domain-specific breakdowns, and the real-plan test still found the governing value in only three of nine numerical-limit items.
The work was financially supported by the Wyoming Department of Transportation’s knowledge-management project under grant RS03225, with Ahmed Abdelaty as principal investigator. The authors state that supplementary materials will be made available upon acceptance.
Paper data and sources
Original title: PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans
Authors: Nabaraj Subedi, Shuvo Dip Datta, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text