A model that combined speech intensity, gaze and perceived interpersonal closeness was better at distinguishing a gap between speakers from an overlap in four-person dialogue than a model that used gaze alone, according to a new preprint. In held-out tests, its mean ROC AUC was 0.76, compared with 0.58 for gaze-only. ROC AUC is a score for how well a system separates two outcomes across different decision thresholds, with higher values indicating better discrimination.
The task was to classify genuine floor transfers, meaning moments when the turn moves from one speaker to another, as either gaps or overlaps. A non-negative timing offset between the outgoing and incoming turns was labeled a gap; a negative offset was labeled an overlap. Backchannels and overlaps within a single speaker’s turn were excluded.
A test built from free conversation
The analysis reused the GaMMA corpus, which contains synchronised audio, gaze and motion recordings from four-person groups recorded in quiet and noisy conditions. It focused on purposively selected native speakers in a controlled setting, speaking freely without an imposed task or turn-taking constraint. No new data were collected for the analysis; the recordings had already been gathered under written informed consent and ethics approval.
For each possible floor-transfer boundary, the model examined one three-second window immediately before it. It combined the outgoing speaker’s speech intensity with gaze-derived signals and a measure of interpersonal closeness, then made a binary prediction: gap or overlap.
The classifier was regularised logistic regression, a model that estimates the likelihood of one of two outcomes while discouraging an overly complicated fit. Its penalty strength was tuned in inner five-fold cross-validation. The final checks used leave-one-group-out and leave-one-condition-out validation across 15 outer folds.
One safeguard addressed information leakage, the accidental use of details from the answer being predicted. Speech-intensity features were restricted to current-speaker, speaker-offset-aligned information and excluded candidate or future-speaker data. The paper reports that including those signals produced unrealistically high AUROCs above 0.90.
Speech carried the larger share of the signal
The combined model’s mean PR AUC was 0.77, versus 0.60 for gaze-only. PR AUC is a companion measure that emphasizes the balance between precision, the share of positive calls that are correct, and recall, the share of relevant cases found. The mean paired gain in ROC AUC was 0.18, with a 95% confidence interval from 0.16 to 0.19. The corresponding PR AUC gain was 0.17, and the Wilcoxon comparison gave p < 0.001.
Ablation tests, which compare versions of a model after changing the inputs, pointed to an uneven contribution from the two signal types. The comparison in which speech intensity was added to gaze-only showed a gain of 0.17 in both ROC AUC and PR AUC. In the other direction, adding gaze to the phon-only baseline showed a PR AUC difference of 0.08 with a plus-or-minus spread of 0.02 and a ROC AUC difference of 0.03. The gain associated with the gaze comparison was positive in every fold, with Wilcoxon p < 0.001.
The advantage held across acoustic conditions. Multimodal AUCs ranged from 0.74 to 0.77, while gaze-only AUCs ranged from 0.56 to 0.59, including noisy conditions. The pattern suggests that gaze supplied a complementary cue to speech intensity when the acoustic setting changed.
At the threshold selected for the best F1 score, the multimodal model had precision of 0.62, recall of 0.80, F1 of 0.69 and accuracy of 0.71. The gaze-only model had precision of 0.50, recall of 1.00, F1 of 0.66 and accuracy of 0.58. At that setting, gaze-only had higher recall, while the combined model had higher precision, F1 and accuracy.
The result has clear boundaries
The model’s direction analysis described phon intensity, the study’s speech-intensity measure, as the dominant multimodal predictor. Higher values were associated with a lower probability of overlap. Gaze-derived predictors retained the direction seen in the gaze-only model, but their relative magnitude was smaller. The result is associational: it does not show that changing a speaker’s intensity or gaze would cause a different turn outcome.
The findings are bounded by the setting. The sample was purposively selected native speakers in a controlled, unscripted four-person conversation format, so the result is not a test of broader speaker populations or different interaction formats. The model also used a single three-second window immediately before each boundary, making the result specific to that interval.
The GaMMA data are publicly available through Zenodo, but the full analysis code is not, because of intellectual-property restrictions; the modeling procedures are described in Methods. The work is a version-one arXiv preprint dated 28 Aug 2026. Funding came from Innovation Fund Denmark through an industrial research collaboration with GN Store Nord.
Paper data and sources
Original title: Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness
Authors: Mark Dourado, Karim Haddad, Henrik G. Hassager, Stefania Serafin
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text