Average recall can hide large differences in how well a vector search system handles individual queries, according to a new benchmark study. The researchers report that an overall target can be met while some query groups still receive much weaker results. Here, recall means how often the search retrieves the nearest matches expected for a query. The finding challenges a tuning approach based mainly on one score averaged across the workload.
The system tested in the study, called RCheck, was designed to respond to that unevenness. It groups queries by their region in embedding space, estimates each group's recall using sampled exact searches, records its performance, and periodically adjusts the search effort assigned to the group when it deviates from the target. In effect, different parts of the workload can receive different search settings.
RCheck was compared with globally tuned pgvector configurations that use one uniform search-effort setting across a workload. The evaluation covered text, face-recognition, vision and knowledge-graph workloads. For each dataset, 90% of the data was indexed and the remaining 10% supplied the query workload.
The average score was not the whole story
Across the tested datasets and index types, the researchers reported a better recall-throughput trade-off for RCheck. At the same target recall, throughput was 7% to 41% higher than with the best globally tuned configuration. At the same throughput, mean recall was 11% to 93% higher, and 8% to 47% more queries met the target.
On GloVe-100, using HNSW, the difference appeared in the distribution of results, not just the average. At the low operating point, the share of queries meeting the reference target rose from 38.7% with pgvector to 46.7% with RCheck. At the medium point, it rose from 55% to 62.6%. At that medium point, mean absolute error from the target fell from 0.35 to 0.18, while shortfall below the target fell from 0.17 to 0.11.
The speed picture on the same workload was also favorable in the reported comparison. Medium-point p99 latency, the time by which 99% of searches had finished, fell from 17.28 milliseconds with pgvector to 6.61 milliseconds with RCheck.
FairFace, another HNSW workload, showed a larger shift in target attainment. At the low operating point, the share meeting target climbed from 24.3% to 38.2%; at medium, it rose from 45.1% to 59.7%. The reported mean absolute error and shortfall fell at both points. At medium, p99 latency was 7.29 milliseconds for RCheck versus 24.13 milliseconds for pgvector.
On CIFAR-100 with IVFFlat, low-point target attainment moved from 10.66% to 57.91%, while medium-point attainment went from 50.8% to 71.8%. Mean absolute error and shortfall also declined at both points. At the medium point, reported tail latency was 6.52 milliseconds for RCheck and 25.61 milliseconds for pgvector.
One workload showed a more mixed result
DBPedia produced a more mixed picture. At a target recall of 0.85, RCheck delivered 23.6% higher throughput. At 225 queries per second, mean recall rose from 0.71 with pgvector to 0.84, a reported 20% gain. At the low operating point, target attainment increased from 31.73% to 42.6%, while mean absolute error fell from 0.31 to 0.24 and shortfall from 0.27 to 0.19. But at the high point, throughput was 41.33% higher while target attainment was 69.8% for RCheck versus 77% for pgvector.
That result is a reminder that the gains were not identical at every operating point. The paper's narrower conclusion is that mean recall alone is an incomplete system objective, and that adapting search effort by query group can reduce recall differences while preserving or improving throughput in the tested setups.
The details of adaptation mattered
The benchmark also tested how RCheck itself should be configured. On GloVe at target recall 0.85, a 15-second monitoring interval gave the best reported balance, with throughput of 310 queries per second, compared with 190 at five seconds and 250 at 60 seconds. The analysis says the shorter interval reduced throughput, while the longer one delayed adaptation.
Group size mattered as well. On GloVe at the same target, five groups reached 280 queries per second and 200 groups reached 200. At a matched throughput of 500 queries per second, the five-group version had mean recall of 0.71, which was 10.13% below the 50-group configuration. The study identifies 50 groups as the best reported balance between local correction and stable estimation.
RCheck's advantage persisted in the paper's parameter checks. With alternative FairFace embeddings, it reached 125 queries per second versus 107 for pgvector, a reported 16.82% improvement. With HNSW's M setting at 12, the comparison was 153 versus 126; at 28, it was 151 versus 125. Across the tested ef_construction settings, throughput improvement at target recall was 20% to 40%.
The added monitoring was not free, but the reported operating costs were limited in the benchmark setup. Creating groups took an average of 260 seconds; assigning a query to a group took less than 0.1 milliseconds, recall estimation used less than 50% of CPU cycles, and runtime adaptation took less than 14 milliseconds.
What the benchmark leaves open
This remains a systems-method evaluation, not a test of how people respond to search results. It covered text, face-recognition, vision and knowledge-graph workloads, with 90% of each dataset indexed and 10% used for queries. The reported comparisons were made across the tested pgvector HNSW and IVFFlat setups. That scope means the evidence speaks to recall distribution, throughput and latency in these benchmarks, not to performance in every vector database or live production workload.
RCheck also adapts at the group level rather than for every individual query, and the benchmark found that the choice of group count changed the recall-throughput balance. The reported gains therefore describe the tested configurations and workloads, rather than a guarantee for every deployment. The document is an arXiv preprint, version 1, dated 25 August 2026.
Paper data and sources
Original title: Analyzing and Reducing Search Quality Differences in Vector Similarity Search
Authors: Sara Mahdizadeh Shahri, Martin Prammer, Jignesh M. Patel, Akshitha Sriraman
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text