A new preprint presents MTDiag, a dataset and evaluation framework for asking whether a large language model, or LLM, can conduct a diagnostic conversation well enough to reach the right diagnosis rather than simply answer a static question. The project examines multi-turn exchanges in which the system's questioning and the evidence elicited are part of the evaluation.
The paper describes dataset construction rather than completed comparative model testing. The authors state that it releases the PatientVector schema, an utterance-generation pipeline and a physician-validated dataset, alongside clinical knowledge-grounded metrics that go beyond diagnostic accuracy.
Three sources, two MIMIC tracks
MTDiag draws on three source types. The paper reports 1.3M DDXPlus cases, 68K MIMIC-IV cases and 415 AJCR case reports after filtering and deduplication. The larger figures are rounded, so the paper does not give one exact combined total.
After hierarchical one-visit-per-patient deduplication, the final MIMIC-IV subset contains 68,346 unique patients. It includes 49,440 patients discharged from the emergency department and 18,906 who were admitted.
The MIMIC material is split into two tracks that expose different levels of information. The emergency-department discharge track uses emergency-department data and a preliminary ED diagnosis. The hospital-admission track includes richer inpatient information and uses the final verified discharge diagnosis.
How the simulated exchange is built
GroundTruth is populated at ingestion and kept out of the dialogue-generation pipeline and the examiner's input.
MTDiag maps the chief complaint, symptoms and other available clinical facts to Unified Medical Language System concept identifiers, or UMLS CUIs. Diagnoses are mapped to ICD-10 codes.
Anchor resolution uses a two-stage local pipeline. medspaCy identifies candidate clinical spans, while QuickUMLS matches those spans against a local UMLS installation and returns a CUI and mapping_confidence score.
UserLM-8B converts structured symptom representations into canonical patient utterances. Those utterances are pre-generated, stored and decoupled from live dialogue.
At dialogue runtime, two agents interact. A MedGemma-powered Patient Orchestrator selects a pre-generated utterance or combination based on the examiner's latest question.
A wider scorecard for the dialogue
The proposed measures include diagnostic accuracy, semantic diagnostic distance, symptom elicitation, precision, recall, reliability and turn efficiency. The framework also names measures for anchoring bias, premature closure, pathognomonic errors and sine qua non errors.
That scorecard is aimed at the conduct of the exchange: what an examiner asks and which symptoms it elicits across turns. The supplied paper does not report a systematic multi-model examiner evaluation, so comparative performance remains unreported.
The questions the paper leaves open
The proposed metrics are not empirically validated in the supplied text, and no mapping accuracy or error rate is reported for the ontology pipeline. The paper therefore presents a measurement framework without establishing how well those measures work in practice.
The ontology layer depends on the UMLS version. The development and working pipeline are tied to the 2025AB release, and rerunning it with another installation may change results or CUI assignments.
Pre-generated utterances set a boundary on the simulated exchange. They are created and stored before the conversation, while the runtime selects a pre-generated utterance or combination in response to the examiner.
Access to the MIMIC-derived portions is also restricted. They require credentialed PhysioNet access and Zero Data Retention processing, and sharing them with third parties is prohibited.
Systematic multi-model examiner evaluation is left to future work. As a result, the paper does not provide diagnostic performance or model rankings, and it does not establish clinical competence or safety.
What the preprint does not establish
For researchers, MTDiag offers a setting for studying the questions an examiner asks and the evidence a patient-side system reveals. For general readers, the central point is narrower: this is a research dataset and evaluation framework, not a clinical deployment study or a demonstration that an LLM is ready to diagnose patients.
The supplied document is an arXiv version 1 preprint dated 25 August 2026. The authors acknowledge BMBF support through the KISSKI AI service center and computing resources under the stated multi-turn medical-dialogue benchmark project.
Paper data and sources
Original title: MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
Authors: Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz et al.
Journal/Repository: Chouayfati, Pia, et al. "MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation." Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2026
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text