A preprint evaluation of Flower Hub found that no aggregation strategy, meaning the method used to combine client updates, consistently dominated across five federated-learning tasks. FedProx was best in most cases, while FedAvg led the finance benchmark and FedAdagrad led legal instruction tuning. FedAvg and FedAvgM were similar, reinforcing the task-dependent pattern.
Flower Hub packages each benchmark as a standardized application with explicit configuration, pinned dependencies and machine-readable metadata. The report says the same benchmark application can run in both simulation and deployment settings without source-code changes, using `flwr run`.
A common test bed
The platform demonstration covers five benchmark tasks: medical image segmentation, financial fraud detection, legal instruction tuning, phishing URL detection and on-device audio tagging.
The datasets vary sharply in the table's reported scale and client count. fed-brats3 is listed with 1.6 K samples and 5 clients; fed-fraud-paysim-banks with 6.4 M and 5 clients; fed-legal with 83.6 K and 5 clients; fed-phishing-urls with 1.1 M and 100 clients; and fed-urbansound8k with 8.7 K and 50 clients.
Each pipeline uses a different model and task measure: a 3D U-Net with Dice score for medical imaging, an MLP with PR-AUC for finance, an LLM with F1 score for legal instruction tuning, a 1D CNN with ROC-AUC for phishing URLs and a CNN with accuracy for audio. These are separate task-specific tests, not one shared score.
The comparison included six aggregation methods: FedAvg, FedProx, FedAvgM, FedAdam, FedAdagrad and FedYogi. For the cross-silo tasks, all five clients participated in each round. Legal ran for 10 rounds; medical and finance ran for 20; phishing sampled 10 of 100 clients per round for 20 rounds; and audio sampled 15 of 50 for 100 rounds.
The winner depended on the workload
On the medical benchmark, FedProx had the highest listed Dice score at 78.58%, just above FedAvg at 78.57%. FedAvg topped the finance benchmark with a PR-AUC of 56.40%, while FedProx recorded 55.21%. In legal instruction tuning, FedAdagrad had the highest listed F1 score at 72.97%, compared with 72.29% for FedProx.
FedProx also had the highest listed result for phishing URL detection, with a ROC-AUC of 98.48% versus 98.45% for FedAvg. In audio tagging, it led accuracy at 52.56%, compared with 48.70% for FedAvg. The reported figures are means from three runs with different random seeds, with standard deviations shown in parentheses.
Scores are only part of the picture
Beyond model scores, the platform reports system measurements under FedAvg simulation. Communication was 3.85 GB for medical imaging versus 0.01 GB for finance. Legal training took 14.63 K seconds and had 42.43 K seconds of latency; URL latency was 244 seconds and audio latency was 2.17 K seconds.
Those figures show why the engineering side of a federated-learning benchmark matters alongside task scores: the workloads carried very different communication and time demands. The system results describe FedAvg simulation, so they do not provide a full system-level comparison of all six aggregation methods.
A baseline with room to test more
The authors describe the work as baseline evaluation, say several important federated-learning settings are not covered, and note that deployment experiments remain limited in scale.
The baseline design leaves open how results would change with more extensive optimization and larger-scale deployment. The paper's central result is narrower: rankings changed by task within the tested setup.
The benchmark applications used in the paper are stated to be publicly available on Flower Hub under named application identifiers. The document is an arXiv preprint dated 25 Aug 2026; the supplied text contains no funding statement.
Paper data and sources
Original title: Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Authors: Yan Gao, Mohammad Naseri, Javier Fernandez-Marques et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text