A training method for spiking language models improved reported language-benchmark scores while avoiding the rollout collapse seen in two comparison settings, according to an arXiv preprint. Across three model scales, the reported performance differences over matched knowledge-distillation checkpoints were 0.8, 1.7 and 2.9 percentage points. In a separate 10-seed stability test, the SpikeOPD run had no collapsed runs after 1,000 updates.
The study focuses on a mismatch between the prefixes used in training and the prefixes a model generates for itself. It asks how that mismatch appears in the model's output policy, meaning its pattern of predicted word choices, and in its internal spiking dynamics. It also asks whether Vanilla OPD can adapt without a sustained rollout-feedback collapse.
How the method was tested
SpikeOPD adapts an offline knowledge-distillation checkpoint with three components: teacher-guided policy correction, matched-prefix policy anchoring and layerwise spike regularization. In practical terms, the setup combines guidance from a teacher, an anchor for the model's behavior on matched prefixes and a check on spike activity across selected layers.
The experiments used matched OPT teacher models and causal BiSpikCLM spiking students at 125 million, 0.35 billion and 1.3 billion parameters. Stage 2 began from an offline checkpoint trained for 4,000 updates, used 480-token FineWeb prompts and generated 32 new tokens at a reported temperature of 1.0. The teacher and reference model were frozen, while only the active spiking model was updated for 500 updates. The language evaluation covered eight zero-shot benchmarks: ARC-Easy and ARC-Challenge, WinoGrande, BoolQ, PIQA, HellaSwag, OpenBookQA and HeadQA.
The scale results used three seeds at each scale. The stability test used 10 matched seeds, and the diagnostic comparing fixed paired prefixes used 256 pairs. The benchmark result was an unweighted mean across the eight tasks, while rollout stability was also assessed through repetition and diversity measures.
Signs of drift appeared inside the model
The diagnostics suggest that the problem was not limited to the words produced at the end of a rollout. For the knowledge-distillation checkpoint, the paired self-versus-corpus gap grew from near zero to about 0.9 in KL units over rollout steps, a measure of how differently the model's predicted word distribution behaved on its own prefixes compared with fixed text. SpikeOPD kept that gap smaller and was also reported to have lower absolute KL divergence from the teacher.
The same pattern appeared in internal signals. The reported hidden-state gap was 0.102 for the knowledge-distillation model, 0.100 for Vanilla OPD and 0.095 for SpikeOPD. The corresponding membrane-potential gaps, another internal state measure, were 3.46, 3.30 and 2.67. The knowledge-distillation gap increased with rollout position, while SpikeOPD stayed lower in the reported diagnostic.
The stress test used collapse thresholds of 0.10 and 0.05, calculated from 20-update means. Both the Small-LR and Clipped controls collapsed in all 10 of their tested runs, with average onset at 527 and 327 updates respectively. The reported SpikeOPD condition had zero collapsed runs in 10 tests over the 1,000-update horizon. That finding indicates stability under this test protocol, not a guarantee that every longer rollout or training run will remain stable.
The gains were modest, and the evidence is still narrow
In a controlled post-training comparison, SpikeOPD was the only listed method reported to raise the knowledge-distillation average, from 34.7% to 35.5%, while keeping rollout statistics close to the initial checkpoint. Across the three scales, the reported three-run averages were 35.5% plus or minus 0.02 percentage points, 40.0% plus or minus 0.19 points, and 44.3% plus or minus 0.12 points. Those plus-or-minus values are sample standard deviations across the three runs.
The reported gains came with no additional analytical compute or downstream-task training. Operation and energy estimates remained slightly below those for the knowledge-distillation model, and firing rates were comparable. At the three scales, the reported firing rates were 0.17 plus or minus 0.006, 0.17 plus or minus 0.000 and 0.19 plus or minus 0.010.
The ablation results point to the reference-policy anchor as the most important of the three listed components in this test. The complete configuration had the highest reported average at 35.5%; removing any component lowered performance, while removing the reference term produced the largest decline, to 31.7%. Across three matched offline starting checkpoints, the reported SpikeOPD gain ranged from 0.6 to 0.8 points, although the maturity of the starting checkpoint still affected absolute performance.
This is a computational methods evaluation. Its efficiency figures are analytical proxies rather than measurements of hardware latency, power or wall-plug energy. Its zero-shot benchmark averages and rollout statistics are proxies for retained language capability and generation stability, rather than direct deployment or human outcomes. The scale averages rely on three runs per scale, and the evidence does not include hardware measurements.
The document is an arXiv version 1 preprint dated 28 August 2026. Its evidence is limited to the tested models, FineWeb-based prompts, zero-shot benchmarks and matched-seed stress tests. The reported stability therefore describes the tested horizon and settings, while questions about other architectures, corpora, longer adaptation and independently reproduced implementations remain open.
Paper data and sources
Original title: SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models
Authors: Enqiao Lu, Xingrui Yu, Yiwei Fu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text