An arXiv preprint reports a sharp split in graph-based AI question answering. After Compositional Restructuring, an answer-preserving structural rewrite, the gold answer was present in 74.0% of retrieved CWQ subgraphs and 94.6% of WebQSP subgraphs, while end-to-end exact-match scores were only 0.68% and 0.49%, respectively.
That metric did not require a character-for-character response. In this evaluation, a generated answer counted as an exact match when it contained the gold Freebase MID, the database identifier for the correct answer, as a substring. The figures show that answer presence in the retrieved graph did not ensure a matching final response under the tested condition.
The test separated the pipeline stages
The study asks which stage of a graph-based question-answering pipeline is associated with failure under adversarial query perturbations. It introduces two answer-preserving tests: Compositional Restructuring, or CR, changes a question's structure, while Relation Synonym Swap, or RS, changes a relation's surface wording. In the RS setup, the entity seeds used to begin retrieval are preserved, and stage isolation lets the analysis examine retrieval and answer generation separately.
The CR and RS variants were required to meet three screens: perplexity below 50, BERTScore similarity above 0.85, and the same Freebase denotation, meaning the same database-level answer.
The CWQ stage accounting
The main evaluation used 3,531 CWQ test questions and 1,639 WebQSP test questions. Its primary outcome was the change in end-to-end exact-match performance between clean and perturbed questions. The analysis also tracked changes in entity linking, retrieved-subgraph drift and answer presence, with bootstrap 95% confidence intervals for the main degradation measure based on 1,000 resamples.
For retrieval comparisons, the GNN-RAG checkpoint was held fixed while only the retrieved subgraph changed across the listed variants. The reported contrasts therefore stayed within one tested checkpoint as the supplied graph information varied.
The CWQ CR contrast was especially stark in the reported stage accounting. With the clean, unperturbed subgraph supplied in fast mode, exact-match performance was 52.76%, compared with 0.68% for the full pipeline. The accounting assigned 52.08 percentage points of the 52.22-point total drop to subgraph failure. That allocation places nearly all of the measured loss at the subgraph stage in that accounting.
The reported ELQ SeedHit measure moved only slightly: degradation under CR and RS was at most 0.3 percentage points from clean.
The relation-wording test had a separate score pattern
Under Relation Synonym Swap, the reported full-pipeline exact-match performance was 20.3% on CWQ and 50.9% on WebQSP. In the clean-subgraph fast-mode condition, the corresponding reported values were 51.69% and 70.59%.
Other comparisons used different setups
Other retrieval configurations were also reported. In GraftNet with oracle GEM seeds, answer presence under CR was 63.3% and CWQ exact-match performance was 29.82%. The deployment-realistic ELQ-plus-cosine configuration's CWQ exact-match figure was 14.98%.
Another reported comparison listed 51.4% CWQ exact-match under CR when predicted relation paths were injected at inference time without fine-tuning. That figure matched or exceeded the relation-path-augmented model in the reported comparison.
EPR-KGQA reported MID-based Hit@1 scores of 59.22% on CWQ and 64.25% on WebQSP under CR. Under RS, its reported figures were 59.76% and 63.09%, respectively. These are Hit@1 results, a separate metric from the end-to-end exact-match scores used in the main comparison.
A separate comparison on MetaQA
The MetaQA comparison recorded smaller reported CR drops: 6.91 percentage points on MetaQA-3 and 0.11 points on MetaQA-2, compared with a 52.22-point CWQ drop from 52.9% clean exact match to 0.68% under CR.
That comparison is not interchangeable with the main result: MetaQA uses a different knowledge base and evaluation setup, so these figures provide context for the reported contrast rather than a direct test of the CWQ evaluation.
What the numbers cover
The authors also report Bonferroni-adjusted 99.8% confidence intervals. The adjustment used an alpha of about 0.002, calculated as 0.05 divided by 24, and the authors state that all five primary conclusions survived the correction.
The core numbers come from a fixed-checkpoint evaluation on the CWQ and WebQSP test sets. They describe the tested setup and stage-isolation comparisons, rather than every knowledge-graph question-answering system or production deployment.
The document is marked as an arXiv version dated 26 Aug 2026, and the paper reports release of the perturbed datasets and evaluation infrastructure.
Paper data and sources
Original title: Query-Side Attacks on GNN-Based KGQA: Tracing Failures from Entity Linking to Answer Generation
Authors: Pankaj Kumar, Subhankar Mishra
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text