A preprint analysis reports that the geometry of token representations changes in a pattern tied to grammatical class. Closed-class items expanded into higher-dimensional neighborhoods earlier and collapsed sooner than open-class items. In a separate test, geometric features distinguished function words from content words, showing that the representations carried information about that broad lexical split.
The researchers followed these changes layer by layer, asking whether geometry reflects lexical class, whether neighborhood reorganization corresponds to expansion or contraction, and whether bidirectional versus causal context integration changes the trajectories. They used two measures of this geometry: PoS-conditional intrinsic dimension, or cID, which estimates how many directions a local token neighborhood occupies, and PoS-conditional information imbalance, or cII, which is based on nearest-neighbor relationships. The measures were calculated separately within documents and averaged across documents.
A grammar signal in local neighborhoods
The main analysis used 539 Pile-10k documents, each between 1,000 and 1,500 words long. It retained the eight most frequent parts-of-speech categories, which accounted for 83% of part-of-speech occurrences. The models were ModernBERT and bigbird-roberta-large as encoders, and Gemma-2-2B and Llama-3.2-3B as decoders.
Across the models, closed-class items showed an earlier rise into higher-dimensional neighborhoods and an earlier collapse than open-class items. The paper links those changes to local neighborhood structure and to the changing relations among words within a sentence. In the authors’ interpretation, the geometry tracks the resolution of the elements that grammatical words combine with.
On the binary task, a classifier using both cID and cII reached 0.878 accuracy, an F1 score of 0.830 and a ROC-AUC of 0.937. These were mean results from 20 repeated runs, with reported variation of plus or minus 0.008, 0.012 and 0.005 respectively. When the geometric features were shuffled, the F1 score was 0.389.
For the seven included parts-of-speech categories, the combined classifier reached 0.529 accuracy, an F1 score of 0.540 and a ROC-AUC of 0.844. The reported variation across 20 runs was plus or minus 0.010, 0.010 and 0.005 respectively. Shuffling the features reduced the F1 score to 0.167.
The pattern depended on the model
The layer-by-layer profiles differed across architectures. In ModernBERT, determiner representations contracted sharply at layer 16 and showed another negative peak at layer 19. BigBird-RoBERTa converged on a shared cID peak at layer 13. Gemma showed peaks for function words, adverbs and verbs at layer 6 and again between layers 11 and 13. Llama’s peaks appeared at layers 9, 15 and 19, while adpositions reached their earliest peak at layer 4.
The comparison was narrow: it examined two encoders and two decoders. The authors note that typology, architecture, pretraining data, model scale and training objective may all affect the geometry observed, so the reported profiles cannot by themselves establish a general pattern for transformer models as a whole.
A second test linked geometry with a meaning-based task
A separate specialty-classification analysis used 928 PubMed abstracts distributed across 15 medical specialties. This downstream analysis was conducted with the two encoder models.
Around layer 19, performance for function words rose more steeply than performance for content words, and the two groups nearly converged. The authors present this pattern as consistent with semantic absorption. The encoder analysis also found a significantly negative association between layer-mean cID and specialty-classification accuracy, strongest for determiners, pronouns and adpositions. That association does not show that geometric contraction caused the classification change.
The open question is how far the pattern travels
The authors restricted the analysis to English and to two encoders and two decoders. They note that typology, architecture, pretraining data, model scale and training objective may affect the observed geometry. The evidence therefore covers the selected English corpus, four specified models and an encoder-only downstream analysis, leaving broader comparisons for future work.
The document is identified as arXiv:2608.25166v1 and dated 25 August 2026. Funding and conflict-of-interest information are not reported in the supplied text.
Paper data and sources
Original title: The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
Authors: Samuele Vallisa, Federico Ravenda, Claudio Palominos et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text