Candidate features inside a Vision Transformer mostly moved toward earlier layers as the model trained, according to a new preprint, while movement toward deeper layers was rare. The analysis also found that deeper layers stabilized earlier and more strongly than earlier ones. That gives a picture of how this model’s internal signals changed over its training trajectory, but it does not establish that the movement caused a change in accuracy or behavior.
The strongest earlier-layer drift reached approximately −2.5 layers in the most affected regions. By contrast, the proportion of tracked features classified as migrating toward deeper layers peaked at 4%, with visible peaks mostly in early checkpoints. The authors describe deeper-layer movement as a minor part of the overall reorganization.
How the researchers followed the signals
The researchers built this map from the model’s CLS-token representations. They used sparse autoencoders to extract candidate sparse features, then compared activation profiles across pairs of training checkpoints and layers. To handle possible differences in feature recovery, the final similarity between two candidates was the maximum of a fixed-SAE measure and an independently trained-SAE score.
The ViT-Tiny model had 12 transformer blocks, a patch size of 16, an embedding dimension of 192 and 12 attention heads. It was trained for 300 epochs, allowing the researchers to trace feature paths within the same architecture across training checkpoints.
The training data came from two settings. ImageNet-1k contributed a random sample of 100,000 training images distributed uniformly across 1,000 classes and a 50,000-image test split. Mixed 10 contributed its entire 77,060-image training split and 2,700-image test split.
The study used operational thresholds to turn feature trajectories into categories. A feature counted as migratory when its tracked layer shift passed one layer, with directional cutoffs above 1 for movement deeper and below −1 for movement earlier. Stability meant net drift below 0.5 layer with stability occupancy of at least 0.7; the spread category used a localization cutoff below 0.5.
The movement was concentrated early
Earlier-layer migration was strongest in earlier layers and earlier training, and the largest signed drift was approximately −2.5 layers. Movement in the opposite direction did not dominate: deeper migration was relatively rare, with a peak of 4% and visible peaks mostly in early checkpoints.
Stability was not evenly distributed across the network. Deeper layers settled earlier and more strongly, while earlier layers also stabilized over training but to a lesser extent. The lifetime patterns added a second layer: some features in earlier layers and early checkpoints persisted for a long time despite being unstable, whereas deeper and intermediate features early in training tended to have shorter lifetimes. Lifetimes late in training appeared stable.
Widely spread features appeared throughout training, but they were concentrated mainly at the beginning and in earlier layers. Combined with the percentage of stable features, that pattern led the authors to report no more significant migration at the end of training.
Feature types did not form migration-based clusters. The analysis found no clustering of feature types by migration, while early checkpoints had fewer class-specific features, which the authors described as a progression over training time.
The data comparison did not isolate scale
The study also compared the two datasets to ask whether dataset scale changed the migration picture. The authors reported that scale did not influence most migration metrics. But the comparison did not isolate image count: the datasets differed in label granularity and test-set size, so it was a contrast between two complete setups rather than a clean test of scale alone.
A useful map with clear limits
These findings are descriptive rather than inferential. The analysis presents patterns through heatmaps, percentages, counts, averages and trajectories, and reports no formal inferential tests or confidence intervals. The reported percentages and drift values should therefore be read as summaries of the studied representations, not as estimates with a stated margin of error.
There are also technical limits to the feature matching. The alignment relies on Spearman rank correlation, which can be sensitive to ties created by zero activations. The sparse-autoencoder procedure does not guarantee that a corresponding feature will be recovered at every other checkpoint-layer pair.
The paper leaves open how much of the pattern belongs to this particular setup. It examined ViT-Tiny with the stated training design and did not test alternatives such as other architectures, random seeds, self-supervised training, fine-tuning or patch-token analyses. The study therefore maps candidate-feature behavior in the reported run; it does not show that migration changes model accuracy or behavior.
The document is an arXiv v1 preprint dated 20 Aug 2026 and says it is an extended version of a CIKM 2026 paper. The authors state that the code is available on GitHub.
The work was supported by Estonian Research Council grant PRG1604 and the Estonian Center of Excellence in Artificial Intelligence, funded by Estonian Ministry of Education and Research grant TK213.
Paper data and sources
Original title: Feature Evolution and Migration during Vision Transformer Training
Authors: Joonas Järve, Halil Ibrahim Aysel, Tarun Khajuria, Meelis Kull
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: 10.1145/3799682.3841045
Original paper · Full text