In an offline test of systems designed to predict when an online discussion may derail, the speech-act model Hparallel ranked first on the reported in-domain Dynamicmean AULC scores when training and testing stayed within the same corpus. It recorded 62.50 for Macro-F1 and 66.85 for AUPRC on WIKI, 65.29 and 67.94 on CMV, and 71.55 and 66.84 on GITHUB. The figures compare models under the study's test setup; they do not show that speech-act supervision prevents hostility, improves moderator decisions or is ready for autonomous moderation.
The study asked whether speech-act information was linked to more robust derailment forecasts when labeled examples were scarce and when training and test domains differed. The authors hypothesized that speech acts might offer a compact pragmatic representation, reducing reliance on lexical noise and improving generalizability. Here, a speech act means the communicative role a sentence plays in a conversation.
What the models saw
The test used three predominantly English conversation corpora, kept separate rather than pooled: 4,188 Wikipedia editor talk-page conversations, 19,578 Reddit ChangeMyView debates, and GitHub issue and pull-request discussions containing 202 derailed and 696 civil dialogues. The preprocessing retained responses posted before derailment, shortened civil training dialogues and limited GitHub discussions to the 10 most recent utterances.
For each sentence, an open-source GPT model assigned speech-act labels in a zero-shot, multi-label task, applying the categories without task-specific hand-labeled examples. The label set contained 50 verbs plus an 'other' category. The labels were used as an auxiliary signal during training, and speech-act extraction was not required when the models generated a forecast.
Hparallel used a two-phase training process: it first learned speech-act detection, then learned derailment forecasting jointly with that information. During evaluation, each model was shown successive prefixes of a dialogue and produced a sequence of probabilities, which was compared with the conversation's ground-truth label. The researchers tested training subsets of 300, 500, 1,000, 2,500 and 11,802 dialogues, and averaged metrics across five random seeds.
Less data, more movement between domains
The reported low-data pattern favored Hparallel. It showed strong performance from 300 to 2,500 training samples and had the highest reported performance across individual forecasting timesteps. At the largest training size, 11,802 samples, the standard text-based language model marginally outperformed Hparallel. The gap narrowed as more data were added, so the ranking difference was not constant across training sizes.
The picture changed when a model trained on one corpus was tested on another. Hparallel led both reported measures from WIKI to CMV, with Macro-F1 of 51.19 and AUPRC of 62.53, and from CMV to WIKI, with scores of 61.49 and 67.14. For transfers into GITHUB, Hsequential led, recording 65.49 and 56.63 from WIKI to GITHUB, and 64.23 and 53.06 from CMV to GITHUB.
GitHub had the greatest measured semantic and lexical deviation among the corpora. The authors reported Hsequential as strongest when transferring into GitHub, while Hparallel performed best between the more similar WIKI and CMV datasets. They interpret this pattern as consistent with speech acts serving as a compact, transferable pragmatic signal, but the corpus distances and forecast results do not isolate a causal mechanism.
The labels brought their own uncertainty
The speech-act labels themselves received a small human check. One author labeled 85 sentences from 30 randomly selected responses, with 10 responses from each dataset. Agreement was moderate across 51 fine-grained classes, with Cohen's kappa of 0.53, but rose to 0.88 when the labels were collapsed into five broader categories.
That result points to a difference between broad communicative categories and finer distinctions. The authors warn that an LLM extractor can reflect cultural, dialectal and community-norm biases, miss indirect or hidden intent, and pass noisy labels into later modeling. The extraction also adds time and computational cost during training, even though it is not needed at inference.
Supplementary checks found Hparallel's reported performance comparable when it used an 18-class taxonomy instead of the 51-class version at 300 training samples. The largest reported discrepancy was 6.91% for Macro-F1 and 2.83% for AUPRC. A separate GraphNLI-SA variant also had higher reported Dynamicmean scores than GraphNLI across the listed train/test configurations, supporting compatibility beyond the hierarchical backbone.
Several boundaries remain. The study is an offline comparison on three predominantly English communities, not a test of deployed moderation or user outcomes. The authors note that the small number of random seeds limits confidence in the stability and statistical power of the rankings; cross-domain Macro-F1 depends on thresholds tuned on source data, and some transfers originating in GitHub were particularly weak.
The document is an arXiv preprint. The authors call for human supervision, manually labeled speech acts, more repeated runs and prospective studies linking forecasts with moderator decisions and user outcomes.
Paper data and sources
Original title: Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecasting
Authors: Angela Yifei Yuan, Christine De Kock, Christopher Leckie
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text