The reported comparisons varied with the model, software backend and execution format. Across NVIDIA Jetson Xavier, Orin and Thor systems, Hydra recorded differences in latency, decode throughput, effective GPU utilization, power and energy per generated token. The results describe the tested configurations, not a universal ranking of chips, backends or formats.
Hydra uses a shared per-prompt timing schema for HuggingFace Transformers and llama.cpp, then fuses those records with hardware telemetry across prefill and decode phases. Prefill is the part that processes the prompt; decode is the part that produces the response token by token. The records keep timing and hardware readings tied to the same prompt.
The speed gap varied across models
At similar 16-bit execution settings, Thor had about 2.8 to 2.9 times the speed of Xavier in the cited comparison. The Thor to Orin gap was roughly 2.9 times for LL-1B and about 1.4 times for GE-7B. The analysis also reported lower end-to-end latency for llama.cpp than for HuggingFace.
The format comparisons showed another distinction. Q4_K_M, a lower-bit execution format in the tested set, showed substantially higher decode throughput across all three generations. For models in the 7B to 8B range, reported throughput was 35 to 44 tokens per second on Thor, 22 to 28 on Orin and 13 to 16 on Xavier. Those are descriptive ranges from the tested configurations, not a result that applies automatically to every model or workload.
Reported quality checks found Q8_0 and Q6_K close to F16. Q4_K_M stayed within 1 point on perplexity, abbreviated PPL, and about 2.4 accuracy points in the worst case.
Overall generation time and token delivery did not always match
For QW-7B on Thor, HuggingFace generation was 33.0 milliseconds while inter-token latency, the time between successive generated tokens, was 63.5 milliseconds. In llama.cpp, generation was 58.5 milliseconds and inter-token latency was 58.6 milliseconds. The measured gap between the two figures was therefore much larger for HuggingFace in this comparison.
The hardware readings also differed between the two backend paths. For QW-1.5B decode on Orin, HuggingFace in bf16, one of the tested 16-bit formats, reported effective GPU utilization of 27.3 and effective DRAM bandwidth of 47.1 gigabytes per second. llama.cpp at F16 reported 97.1 utilization and 96.2 gigabytes per second. The figures are means with standard deviations over prompts.
The length sweeps contrasted prompt delay and decode rate
Time to first token, or TTFT, was about 5.0 to 5.7 times higher in the 5,000-token input condition than in the 1,000-token condition. In the output-budget comparison, the 1,000-token and 5,000-token conditions differed by only a few percent in steady-state decode throughput in most cases. These are descriptive comparisons from the tested sweep.
Energy and power followed different patterns. For QW-7B decode, F16 used 4068 millijoules per token on Orin and 3000 on Thor; Q4 used 1867 and 1493 millijoules per token, respectively. In the same F16 comparison, Orin power was 43.5 watts and Thor power was 51.2 watts, even though Thor's energy-per-token figure was lower. The values are means with standard deviations over prompts.
Power and temperature did not vary monotonically with quantization bit-width. Q6_K often drew more power and ran hotter than both Q8_0 and Q4_K_M, despite using fewer bits than Q8_0.
A defined edge test bed
The main sweep combined 3 SoC generations, 13 models and 5 execution formats, yielding 195 comparison cells. Of those, 190 contributed roughly 103,000 per-prompt records. The full released corpus contains 107,110 per-prompt records in 286 unified CSV files, including the 190 main-corpus configurations and the length-sensitivity sweeps.
All runs used batch size 1, greedy token-by-token generation and KV-cache reuse, meaning the stored context was reused during generation. The tests used the integrated GPU through CUDA, while llama.cpp used full GPU offload. Configurations that could not allocate under that policy were marked failed rather than partially offloaded.
The reported trends are specific to the three NVIDIA Jetson SoCs and their software stack, telemetry interfaces, CUDA path and power-management policies. The evidence does not establish a universal ranking of chips, backends or execution formats.
The authors released the Hydra code and complete measurement corpus through a public GitHub repository and a Zenodo archival copy. The front matter states that the paper was accepted at IEEE IISWC 2026.
For comparisons of edge language-model deployments, overall latency or throughput alone can miss differences between prompt processing and token generation, backend timing, hardware utilization, power and energy. Hydra's records put those measures on the same per-prompt, phase-aware basis, while the tested results show that Q4_K_M throughput, quantization quality and power behavior can point in different directions.
Paper data and sources
Original title: Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels
Authors: Amir Taherin, Sana Taghipour Anvari, Charles Amante et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text