An AI framework designed to answer hate speech with category-specific counterspeech reported higher category accuracy and factuality than comparator systems, along with lower toxicity and greater output diversity. Called FIRE, it was evaluated on benchmark data, and the study did not assess long-term impact or conflict escalation. The findings therefore describe performance on the reported tests, not how the system would affect online conversations over time.
The logic behind the replies
FIRE separates analysis from response writing. Its Hatespeech Analyst sorts an input into five categories: misinformation, stereotype, conspiracy, dehumanizing language and non-factual content. It identifies the target and the reasoning behind the attack, and can trigger a web search when a factual claim needs checking. A separate Counter-speech Generator then uses that analysis, retrieved examples and evidence to produce a response adapted to the category. The agents are not fine-tuned.
The benchmark was FactualCS, a dataset of 4,784 hate-counterspeech pairs covering 14 target communities. It adds dense annotations for hate categories, reasoning traces and evidence mappings. Its data were divided into 3,912 training instances, 383 validation instances and 489 test instances using stratified splits. The distribution is specific to this benchmark, so the figures describe the data used here rather than the wider universe of hate speech.
Six annotators manually labeled the main-phase hate type, reasoning, target group, query and evidence, with agreement reported at around 91.55%. The pipeline used Gemini-2.0-Flash to generate counterspeech from few-shot examples drawn from a 500-sample gold-standard dataset.
A separate manual audit of 100 retrieved evidence instances found that 92% came from institutional or mainstream informational sources. Because the audit covered only the reported sample, it does not establish that every search result was reliable.
What the benchmark recorded
To judge whether a reply matched the intended category, the study used Category Accuracy, or CatAcc. A response counted as correct only when three elements matched: the dataset's gold label, the predicted hate category and the detected counter-strategy. Factual Score, or FSc, used BART-MNLI to check whether claims in generated responses were supported by retrieved gold evidence. In plain terms, the test rewarded replies whose category and strategy matched the labels and whose claims aligned with the evidence retrieved for them.
On the reported category and factuality measures, FIRE posted CatAcc of 0.702 and FSc of 0.969. The paper gives relative improvements of 11.1% and 12.2%, respectively, although its analysis does not reconcile the FSc percentage basis with the displayed table values. FIRE also reported the highest BERTScore, at 0.886, METEOR at 0.247 and CoSIM at 0.574, with competitive ROUGE scores of 0.294 for R1, 0.071 for R2 and 0.197 for RL. No confidence intervals were reported for those alignment estimates.
On safety and variety measures, FIRE's reported Toxicity score was 0.016, described as 11.1% lower than the strongest baseline. Its Repetition Rate was 0.118, 7.1% lower, while Novelty was 0.730, 9.1% higher, and Diversity was 0.873, 1.3% higher. These are scores from automatic evaluation of generated text. They describe the benchmark comparison, while confidence intervals for the metric differences were not reported.
What changed when parts were removed
Component comparisons pointed in the same direction. With web search removed, FSc was 0.832 instead of 0.969 and CatAcc was 0.664 instead of 0.702. Without memory, CatAcc was 0.599, CoSIM was 0.490 instead of 0.574, BERTScore was 0.874 instead of 0.886 and Repetition Rate was 0.152 instead of 0.118. The paper did not report confidence intervals for these ablation estimates.
In the no-HSA condition, the paper reported drops of 36.0% in FSc and 27.7% in CatAcc. Taken together, the ablations show that the full configuration scored better on the listed measures than versions missing web search, memory or HSA. They do not by themselves show how each component would perform across other datasets or languages.
Human judgments and hardware
In a separate human evaluation, 30 expert annotators aged 20 to 35 compared FIRE with LLaMA-3.1-8B-Instruct SFT on a randomly sampled subset with balanced category coverage. FIRE's reported win rates were 0.91 for ICS, 0.89 for Ad, 0.86 for CoRl and 0.93 for ArgE. The study does not report how many response pairs were judged or confidence intervals for those win rates, and the result does not establish preference among broader user populations.
FIRE's other notable result concerned hardware. Sequential operation loaded one 1.7B model at a time, and the paper reports peak VRAM of roughly 4GB, compared with approximately 16GB typical for 7B to 8B models. That is reported as a 75% reduction in peak memory, with end-to-end latency of 2.5 to 3.0 seconds. The paper does not report memory or latency variability, so these figures should be read as measurements from the reported setup.
A narrow test window
Paired two-tailed tests on identical test instances used p < 0.05 as the significance threshold. The authors reported significant SFT gains for Novelty, Category Accuracy and FactualScore, with t-statistics of 12.72, 6.84 and 14.92, respectively. They also reported significant FS gains for Category Accuracy and FactualScore, with t-statistics of 9.27 and 18.63. The analysis did not report confidence intervals or a multiple-testing correction.
Those findings remain bounded by the study's scope. FactualCS covers five categories and 14 target communities, while the evaluation is English-only, so the results may not carry over to other forms of hate, languages, cultures or intersectional attacks. The authors also warn that compact agents may provide less reasoning depth for highly ambiguous language and that external search failures can propagate errors into generated replies. The study measured immediate response quality rather than long-term conversational impact or conflict escalation. Broader real-world performance therefore remains untested.
Paper data and sources
Original title: Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation
Authors: Sujoy Nath, Aswini Kumar, Tanmoy Chakraborty
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text