Energy per generated AI token rose much more with a longer context in the two models labelled MHA than in the two models using GQA-based configurations, according to a four-model analysis. From a context of 128 tokens to 1,800 tokens, energy per token increased 17.92% for OPT-1.3B and 16.98% for Phi-3 Mini. It increased 3.62% for Gemma-2B and 3.32% for Mistral-7B. Gemma-2B uses GQA, while Mistral-7B combines GQA with SWA. The figures cover the decode phase, the part of inference included in the energy evaluation.
The pattern concerns scaling with context, not simply which model used the least energy. At 128 tokens, the shortest tested context, OPT-1.3B used 1.5960 joules per generated token. Gemma-2B used 2.3310 joules, Phi-3 Mini 3.7009 joules and Mistral-7B 4.8905 joules. Those measurements put the four models on a wide range of baseline decode-energy levels before the context was extended.
Longer contexts, meanwhile, barely changed token-level latency in the same tests. OPT-1.3B ranged from 10.84 to 11.04 milliseconds per token across contexts. Phi-3 Mini ranged from 24.41 to 24.46 ms, Gemma-2B from 25.91 to 26.17 ms and Mistral-7B from 25.52 to 26.14 ms. Energy scaling and speed therefore moved differently: the energy measure changed more clearly than latency per token.
The energy curve shifted inside a response
The researchers also examined what happened inside a generation, comparing token positions 50 and 950 as the key-value cache grew. In the two MHA models, the within-sequence drift was larger. OPT-1.3B's drift ranged from 9.64% at context 128 to 2.81% at context 1,024. Phi-3 Mini's ranged from 5.41% to 6.43% across the tested contexts.
The GQA-based profiles were close to flat by comparison. Gemma-2B's drift figures were 2.12%, 0.99% and negative 0.32% across the tested context lengths. Mistral-7B's were positive 1.02%, negative 0.69% and negative 0.83%. The analysis notes that Mistral's values fell within measurement variability, so the small changes should not be read as a steady rise.
Batching changed the per-token cost
Batching was associated with a much larger shift in the per-token energy measure. When batch size increased from 1 to 8 requests, energy per generated token was more than 80% lower in nearly all configurations. For the MHA models, the reduction was approximately 84% at context 128, falling to approximately 67% at context 1,800. Gemma-2B and Mistral-7B kept reductions above 82% across contexts.
The test evaluated four representative open-source language models and varied the model, attention configuration, context length, token position during generation and batch size. It used 16-bit FP16 greedy decoding, used 200 generated tokens for the context-length, architecture and batching comparisons, and used a 100-token chunk for the token-position comparison. Context lengths were 128, 512, 1,024 and 1,800 tokens, while batches contained 1, 2, 4 or 8 requests. The three decode-efficiency metrics were energy per generated token, latency per token and latency per request.
Energy was calculated from GPU readings taken immediately before decode and after the final output token, using the difference between the two cumulative readings. Each configuration was repeated 10 times on a dedicated GPU node, with a 20-millisecond settling period before and after each measurement. The reported energy is therefore a decode-phase GPU measure, not a whole-system or environmental total.
The comparison has a narrow reach
That pattern needs a careful reading. Only four selected models were compared, and each model came with its own attention setup. Mistral-7B also combines GQA with SWA. The comparison therefore cannot isolate an independent effect of attention architecture from differences between the models or determine the separate contribution of GQA and SWA. It shows an association among the tested configurations, not a causal effect.
The scope is also limited by the workload. The study used 16-bit FP16 greedy decoding and tested only context lengths from 128 to 1,800 tokens and batch sizes from 1 to 8. Its percentages therefore do not establish the same pattern for other precisions, decoding strategies, contexts or batch sizes. Because the energy evaluation excluded prefill, it also does not describe the energy of all stages of inference.
Repeated measurements add a consistency check, but the analysis did not supply the standard deviations, coefficients of variation or confidence intervals behind the stability assessment. It also did not report formal hypothesis tests or a sample-size rationale. The reported percentages are best read as results under this test setup, with their broader uncertainty still open.
For teams running long-context workloads, the results leave two practical questions: whether an attention setup keeps energy flatter as context grows, and how much batching can reduce per-token energy. The study's four-model scope leaves open how the pattern would look in other model families and implementations, especially in comparisons that separate architecture from model size.
Paper data and sources
Original title: Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
Authors: Molka Chkir, Syed Muhammad Danish, Jos Höll, Arghavan Asad
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text