An end-to-end audio-language model that groups speech recordings under a chosen instruction scored higher than every comparison system in a benchmark, including tests based on linguistic content, emotion and background noise. AudioLens-R1 achieved an overall Adjusted Rand Index (ARI) of 44.77 and a V-measure of 73.43, the best overall performance on both reported metrics. The study compares the model’s proposed groups with benchmark partitions, so the scores show agreement with the tested labels rather than downstream retrieval or user benefit.
The lead held across the benchmark
Against GPT-audio-1.5, identified in the analysis as the strongest competing system, AudioLens-R1 was ahead by 12.99 absolute points on ARI and 11.62 points on V-measure. It recorded the best ARI in all 12 clustering settings and the best V-measure in 11 of them.
The question is part of the test
AudioLens-Bench defines multi-perspective clustering as a two-part job. Given a set of recordings and a natural-language perspective, the system must infer how many clusters are needed and decide which recording belongs in each one. The perspective can concern linguistic content or paralinguistic information, such as emotion or background noise. The benchmark divides perspectives into held-in cases seen during training and held-out cases whose instructions, categories and instances are reserved for evaluation.
Synthetic voices, controlled conditions
The benchmark draws on four complementary corpora: ECHR legal cases, S&P 500 annual-report excerpts, Banking77 banking-intent queries and MultiWOZ task-oriented dialogues. To build instances, the researchers expanded short queries, compressed longer documents into summaries designed to preserve evidence, or reused native dialogues. They then validated the labels and synthesized the material with text-to-speech, adding controlled paralinguistic attributes. The supervised fine-tuning set contained 1,876 episodes from 2,008 unique recordings. Episodes averaged 5.64 clips and 3.37 gold clusters, with up to nine clusters.
What the benchmark leaves open
That construction is central to how the findings should be read. The test used synthesized speech and controlled attributes rather than naturally recorded audio, and it covered only the four source corpora and the perspectives selected for the benchmark. The reported advantage therefore does not establish performance on real-world recordings, arbitrary unseen perspectives or new domains. The paper reports benchmark scores rather than downstream retrieval, discovery or user utility.
Training was tuned for the task
AudioLens-R1 is an end-to-end large audio-language model trained first with reasoning distillation and then with preference optimization. The latter stage used a screened on-policy preference set of 1,075 training pairs from held-in perspectives. The study compared the full system with ASR-based pipelines that combine text representations with classical clustering, ASR-based LLM clustering and native audio-language baselines. Embedding methods received the gold cluster count, while the LLM and audio-language systems had to infer it.
The combined recipe made the biggest difference
Within the reported ablations, answer-only supervised fine-tuning produced an overall ARI of 34.83. Reasoning distillation alone raised ARI to 35.97, a gain of 1.14 points, but added only 0.28 V-measure points. Preference optimization alone reached ARI 37.07 and V-measure 67.39, gains of 2.24 and 1.71 points. These are score comparisons between the study’s training setups, so they do not isolate every possible reason for the gap.
With both stages, the combined condition reached 44.77 ARI and 73.43 V-measure. That was 9.94 and 7.75 points above answer-only supervised fine-tuning, and 7.70 and 6.04 points above preference optimization alone. The authors interpret the pattern as evidence that native audio modeling, task-specific reasoning distillation and preference optimization improved clustering-oriented reasoning and the use of paralinguistic cues.
Strongest results came on acoustic perspectives
AudioLens-R1 led the macro-average results for background noise, emotion and linguistic reasoning. Its ARI/V-measure pair was 41.45/60.35 for background noise, compared with 7.53/32.78 for the strongest corresponding baseline. For emotion, the figures were 39.00/69.00 versus 2.40/37.63. These remain benchmark results on synthesized speech, so the perspective-level gaps do not settle how the model would perform on naturally recorded audio.
Paper data and sources
Original title: AudioLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models
Authors: Wenjun Huang, Qiaosong Chu, Tiger Shao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text