The central finding is a split between the geometry attention actually exposes and the coordinates used to store it. The preprint’s worst-case theory says the rank needed to approximate normalized softmax attention can grow with a power set by one-half the retained interaction dimension. A separate audit of learned heads found that native query and key geometry added no detectable held-out information once the study’s B1 baseline was included.
A rank target tied to outputs
The study defines approximation rank as the smallest rank among unrestricted matrices that remain within a chosen maximum row-wise absolute-sum error of the original attention matrix. This is an output-preserving target: the condition is exactly equivalent to requiring uniformly bounded error over all bounded value outputs. The measure therefore asks how much rank can be removed without exceeding the specified worst-case output error.
What softmax can actually see
Softmax removes row-constant logits, so the visible interaction is a centered matrix determined by the attention matrix itself. The exact visible interaction dimension is the rank of that centered matrix. This separates the interaction that normalized attention can see from the particular query and key coordinates used to represent it.
That distinction matters because an invertible reciprocal transformation of the query and key factors can leave their product, logits and normalized attention unchanged while changing Euclidean radii and SVD-selected directions. Native geometry is therefore representation-specific rather than an invariant property of the attention pattern.
A sharp law, with conditions
The proposed intrinsic profile combines a projective residual, which measures what remains outside a retained interaction, with the factor-radius size of that same interaction. Because it is built from the visible interaction rather than an arbitrary coordinate factorization, the profile is gauge-invariant and provides theorem-level upper control. Its dimension-dependent cover constants are unknown, however, so the profile is not a direct finite numerical certificate.
A finite-grid construction makes the size exponent sharp in a narrower sense. For a fixed interaction dimension and error tolerance, its approximation rank grows as the grid side raised to that fixed dimension, equivalently following a power set by one-half the intrinsic size. The construction establishes sharpness under those fixed conditions, but not joint sharpness in size, residual margin and tolerance.
The geometry of the allowed support changes the law as well. For fixed dimension and tolerance, spherical support has an exponent equal to half of one less than the dimension, while full-ball support has an exponent equal to half the dimension. The full-ball lower result applies only when a token-count condition involving an exponential threshold is met and the scale parameter is sufficiently large.
The empirical check stayed close to the target
The learned-head audit covered 2,302 cells from BERT-base, GPT-2 and Qwen2.5 common-prefix blocks, along with two ViT-Base checkpoints. It used 64 text sources and 64 ImageNet-1k validation images. A separate final certificate pass covered 3,326 cells at three tolerances and produced 9,978 lower-and-upper brackets. The empirical panel was limited to these listed checkpoints and supports.
The brackets were not treated as exact answers by default. The certificate hierarchy paired checked constructive upper witnesses with lower witnesses over unrestricted real competitors, using tests that included Frobenius-tail, truncated-pseudoinverse, residual-aware and decoded-inverse criteria. It explicitly did not treat failure to find an upper witness as a lower bound. In the final checks, no bracket failed, none of 458 targeted rank-reduction attacks succeeded, and all 119 sampled upper constructions passed direct checks. Those checks support bracket soundness but do not identify the optimum when the lower and upper endpoints differ.
At an error tolerance of 0.25, direct resolution varied sharply across families. GPT-2’s common-prefix panel resolved 99.7% of cells, compared with 77.1% for Qwen2.5, 59.0% for ViT-Base IN1K, 61.9% for ViT-Base IN21K and 42.8% for BERT native. The overall identification gate did not pass because BERT resolved 219 of 512 cells. The reported upper endpoints were constructive budgets rather than minimum ranks, so a bracket that did not close left the exact optimum unresolved.
Native geometry fails the main add-on test
The primary held-out comparison asked whether adding the native-coordinate geometry block G to the B1 feature block improved prediction of the normalized constructive upper budget. The response was the constructive budget at error tolerance 0.25 divided by the smaller matrix dimension. The analysis used per-family ridge models, source-disjoint outer folds, grouped inner tuning and source-cluster bootstrap refits.
With equal weighting across four primary families, the mean absolute error difference, calculated as B1 minus B1+G, was -0.00005, with a 95% interval from -0.00060 to +0.00025. The relative gain was -0.0035, with a 95% interval from -0.0422 to +0.0179, and no family interval excluded zero. In plain terms, the native geometry block showed no detectable incremental held-out information beyond B1. The absolute-scale null has a headroom qualification because B1 error was already small in several families.
Against the weaker B0 reference, however, the native block showed a positive aggregate signal. The reported held-out gain was 0.0279, with a 95% interval from 0.0254 to 0.0300, and the relative gain was about 0.36. Within-head permutation retained most of that gain, consistent with coarse head-level structure. This result is descriptive and does not establish causal predictive value.
A coordinate caveat remains
The native-coordinate result also comes with a gauge caveat. Diagonal query and key gauges with condition ratios of 4 and 16 left the attention matrix unchanged but moved frozen native predictors beyond the study’s resolution scale. The intrinsic kappa/q descriptor was not extracted on the learned-head panel, so the empirical test did not evaluate that theorem-level descriptor directly.
The theoretical assumptions are narrow: finite logits on a common positive support, without coverage of a full row-dependent causal mask. The empirical panel covers one BERT encoder, GPT-2, Qwen2.5-1.5B and two ViT-Base checkpoints. The claims concern representation rank, not runtime, memory, sparsity, downstream accuracy or computational hardness. Taken together, the preprint offers a precise output-preserving rank target and a bounded audit of learned heads, rather than a general performance claim about attention systems.
Paper data and sources
Original title: The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
Authors: Yuhe Sui, Jianing Zhang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text