An arXiv preprint reports a median prediction error below 15% for a lightweight model designed to estimate how long transformer training will take, how much work each GPU or training island can deliver, what the run may cost and where bottlenecks may appear.
A map of the whole training run
ShardMeter is intended to cover sharded, distributed and decentralized transformer training. Its target outputs include end-to-end runtime, per-GPU and per-island throughput, wall-clock time, monetary cost and bottleneck location, using model and hardware characteristics as inputs.
To produce those estimates, the model uses regression models for computation and communication operation costs. The predictions are assembled into execution graphs that record dependencies and possible overlap, allowing critical-path analysis of the work that controls the full training step.
Built from measured operations
The benchmark catalog set parameter boundaries from observed LLM values and heuristically selected up to 200 configurations. This created a bounded set of cases for the model to analyze rather than an exhaustive sweep of every possible transformer design.
For computation, the study varied key transformer hyperparameters, ran 10 training iterations per configuration and took execution times from PyTorch Profiler traces. For communication, it measured collective-operation time within and between nodes for payloads ranging from 1 to 150 MiB.
Single-node experiments used a workstation with two CPUs and four A100 GPUs, each with 80 GB of HBM2. Distributed nodes used four A100 GPUs and four HDR-200 InfiniBand cards.
The computation validation covered five named workloads: SmoLM 360M, Llama 3.2 1B, Phi-1.5 1.3B, Gemma 2B and Yi 6B.
The error varied by operation
For communication, mean absolute percentage error, or MAPE, was 14.99% for all_gather, 13.1% for reduce_scatter and 2.14% for all_reduce. MAPE expresses the size of a prediction miss as a percentage, making the three operation-level results directly comparable.
Computation predictions varied more by case. MAPE ranged from 5% to 29%, while median error for every operation and target device stayed below 13% in the evaluated set.
At the end-to-end level, all runtime predictions had error rates below 13% when the largest batch that fit in memory was used on both tested platforms. The result is therefore tied to those platforms and that memory-fitting batch choice, rather than being a general performance guarantee.
The reported MAPE figures are summary accuracy measures; the analysis did not provide confidence intervals or variability estimates alongside them.
The bottleneck can move
The execution-graph analysis found a clear change in the dominant constraint. Small-batch runs were communication-dominated, while larger batches became progressively compute-dominated. In practical terms, moving data set the pace at one end of the tested range, while arithmetic work increasingly did so as the batch grew.
That transition is not fixed. The analysis says it depends on hardware, model architecture, hyperparameters and system configuration, so the bottleneck identified in one setup should not be treated as a universal rule.
A slower network changes the schedule
The paper also used an execution-graph simulation to examine Streaming DiLoCo under a slower-network scenario. Training halted while waiting for the outer step, showing how a synchronization point can interrupt the simulated run.
Changing the schedule altered the illustrated behavior. Doubling the number of fragments increased the example from 6 to 12 iterations, while increasing the step size from 2 to 3 left more buffer for the outer step. The analysis notes that such changes may affect loss or replica drift, and this result remains a simulation rather than a reported live deployment.
The optimizer makes cost visible
The deployment search was applied to one simulated Llama-8B case with a global batch size of 2 million tokens across at most 10 islands. Cluster A was defined as 10 nodes with 8 H200 GPUs per node, priced at $3.59 per GPU-hour. Cluster B used 10 nodes with 8 A100 GPUs per node at $1.19 per GPU-hour.
The result was a trade-off between throughput and spending. Accepting 99% of maximum achievable throughput corresponded to a 21.69% lower cost than the throughput-optimal configuration.
That figure belongs to one simulated Llama-8B deployment study. The analysis also notes that its cost model used online market prices without a reported price-sensitivity analysis, so the comparison is best read as a scenario-specific estimate.
The test boundary matters
The accuracy results sit inside the hardware and workload boundary used for validation: the study relied on specified A100 and H200 systems and the named transformer workloads, while the decentralized cost result came from a simulated deployment. The evidence describes measured-runtime validation and scenario analysis within those settings.
ShardMeter's reported outputs are estimates of runtime, throughput, cost and bottleneck behavior. Its most concrete numerical findings are the operation-level error rates, the end-to-end errors below 13% under the stated batch condition, and the simulated cost-throughput trade-off.
Paper data and sources
Original title: ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork
Authors: Tim Beringer, Patrick Diem, Felix Wolf, Arya Mazaheri
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text