An arXiv preprint argues that the difficulty of compressing normalized softmax attention depends on the geometry of the space occupied by the query and key vectors. Its central question is which geometry controls rank complexity. The study uses maximum-row L1 approximation rank, meaning the smallest rank that keeps every row of the attention matrix within a chosen total absolute-error tolerance. It also gives this measure an exact operator interpretation for arbitrary normed value spaces.
Two geometries, two laws
The clearest result comes from the worst-case setting in which queries and keys lie on a sphere. For fixed ambient dimension d of at least 2 and an error tolerance ε between 0 and 1, the approximation rank is capped by n, the number of tokens, and follows a temperature term with exponent (d−1)/2. In plain terms, it grows like the smaller of the token count and one plus inverse temperature β, raised to that power, up to constants that can depend on d and ε. These are mathematical worst-case bounds for the allowed configurations, not estimates from a sampled population.
Full-ball support produces a different law. Here the exponent is d/2, but only under an explicit large-token condition: β must be above a threshold depending on dimension and error, and n must be at least a dimension-dependent constant times e raised to β/8. In that regime, the worst-case rank is comparable to β raised to d/2, with constants depending on dimension and error. The result is a worst-case state-capacity theorem, not an ordinary prediction of how rank changes with context length.
What the softmax can actually see
The authors then narrow the question from the whole ambient space to what softmax can actually distinguish. A direction that adds the same scalar to every logit in a row is invisible after softmax normalization, so the analysis removes such directions and counts the remaining visible query-key interactions. If that visible dimension is r, the upper bound is capped by the token count and grows, up to a dimension-dependent constant, like one plus the shape parameter divided by ε squared, raised to r/2. The shape parameter is inverse temperature multiplied by the projected query and key radii. This separates two roles: support shape controls worst-case temperature scaling, while visible interaction geometry describes the complexity exposed by an individual head.
An upper bound can be loose, so the paper also supplies lower-bound constructions. For every fixed visible dimension r of at least 1 and error tolerance between 0 and 1, bounded examples at inverse temperatures above a threshold use a token count that scales as β raised to r/2. Their approximation rank is at least a constant times the same quantity, while the shape parameter is of order β. The authors call the matching r/2 exponent minimax sharp, meaning the general dependence is supported from both sides.
The framework also allows the interaction subspace to be approximate. If a candidate subspace W leaves a residual output tolerance τ_W after the omitted interaction is accounted for, that residual uses part of the target tolerance ε. The projected cover must therefore work with the remaining budget ε−τ_W, which requires τ_W<ε, and the resulting rank is still capped by n. If the visible dimension is zero or the projected shape parameter is zero, rank one is valid when τ_W≤ε.
A useful signal in BERT, with clear limits
Tests on synthetic constructions broadly followed the predicted exponents. Sphere fits produced slopes of 0.504, 1.012 and 1.381, against reference slopes of 0.5, 1 and 1.5. Full-ball fits produced 0.476, 0.951, 1.427 and 1.903, against the corresponding d/2 predictions. The largest grouped full-ball point was explicitly described as a state-capacity check, representing 17,850,625 message states and an effective token count of about 1,088,946.
The learned-representation check was a fixed calibration of 84 BERT-base attention heads. It covered sequence lengths of 64, 128 and 256 tokens, five temperature multipliers, and error levels of 0.15, 0.25 and 0.35. Reconstructed attention agreed with the model to a reported discrepancy of 1.57 × 10−6.
At ε=0.25, 41.9% of head-temperature cells had a strictly smaller singular-value-decomposition, or SVD, effective dimension when the internal tolerance was set to ε/3, although the reduction was usually modest. The exact visible dimensions in this calibration were 63–64. The SVD measure is the smallest candidate interaction dimension whose width-based residual is within a selected output-error tolerance; it is not explained variance.
At multiplier one, the effective dimension had reported Spearman associations of 0.574 with the attention-SVD rank upper certificate and 0.606 with the representative-row upper certificate. The comparison concerns finite constructive upper certificates, not the unknown optimum approximation rank.
An aggregate multiplier-one summary at ε=0.25 put the median SVD effective dimension at 64, the median attention-SVD certificate at 48, the median representative-row certificate at 87.5 and the median top-k budget at 45. These figures summarize the fixed 84-head calibration, do not specify the parameter tuple for any individual head, and were not intended as population-level inference.
The paper's scope is consequently narrow but concrete: it offers worst-case geometric laws and per-head approximation certificates, while the learned-head evidence remains a fixed BERT-base calibration rather than a universal law. The document is an arXiv preprint, version 1, dated 28 August 2026.
Paper data and sources
Original title: The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
Authors: Yuhe Sui, Jianing Zhang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text