A preprint reports a narrow but notable result: in Qwen3-8B, a J-lens variant that kept only the highest-energy Jacobian position pairs had higher reported readout scores. The reported gains appeared when the retained fraction was the top 10% to 20% by energy. The study tested six tasks: association, multihop reasoning, multilingual evaluation, ordering operations, poetry and typos.
The mathematics behind the lens
The J-lens is presented as an averaged first-order transfer operator. In ordinary language, it uses a Jacobian, a local linear approximation, to connect changes in an intermediate activation with later readouts. The sample-averaged Jacobian is then presented as the globally optimal linear approximation under a minimum-mean-squared-error criterion.
That mathematical result comes with a condition. Under Gaussian inputs and the stated regularity assumptions, the average Jacobian is reported to equal the population least-squares slope, linking the J-lens to an optimal linear readout. The analysis also reports that nonlinear and non-Gaussian errors decrease as the source layer approaches the target layer, and that empirical overall-error measurements support the theoretical account.
A signal concentrated in a few positions
The energy analysis gives the method a more concrete shape. Total Jacobian energy decreases with depth, while energy across position pairs is heavy-tailed and increasingly concentrated in a small fraction of pairs. For the structural visualization, the authors displayed the 400 highest-energy position pairs per layer, approximately the top 13%, and hid the rest.
Those pairs were grouped into two positional patterns. Diagonal structure was interpreted as short-horizon token prediction. Horizontal or vertical patterns were associated with highly sensitive positions at critical locations and with sparse-concept readout.
Where the scores moved
To compare the variants, the analysis used two measures. Short-horizon layers, or SHL, captures whether the top-1 readout in trailing layers matches the true next token. Intermediate-concept recall rate, or ICR, captures whether an annotated concept token appears in a position-layer cell's top-10 readout. All J-lens variants used the same configuration and the same 200 construction examples, while vanilla denotes the original J-lens.
In the reported averages, the top-20% energy filter had SHL of 1.00 and ICR of 22.14, compared with 0.94 and 18.71 for vanilla. The top-10% filter had SHL of 1.00 and ICR of 22.00. At the top 1%, the averages were 0.98 for SHL and 16.45 for ICR.
The two modes stayed linked
The authors also tried to separate the positional patterns with shape-specific masks aimed at diagonal and horizontal or vertical concentration. One variant zeroed diagonal entries and retained the rest for sparse-concept readout.
The separation did not hold cleanly in the reported averages. The no-diagonal variant had SHL 0.63 and ICR 15.02, below vanilla's 0.94 and 18.71, while the diagonal variant had SHL 1.00 and ICR 17.07. The authors report that diagonal filtering weakened but did not eliminate next-token prediction and also lowered sparse-concept performance, indicating that the two capabilities remained coupled.
An early result
The authors interpret the J-lens through these two modes and regard the energy-filtering comparisons as support for that view. They also state that the work has limitations, remains in progress and does not represent final results.
Paper data and sources
Original title: Short Horizons and Sparse Concepts: a Mathematical View of the Readout in the J-lens
Authors: Shi-Qi Yan, Kai-Xuan Ding, Chao-Hong Tan et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text