Automated tools for producing model cards—the structured documents that accompany AI models—can preserve most of what is already there, but generating a new card from repository files remains error-prone, according to an arXiv version-1 preprint. The study examined two large-language-model workflows: MCTidy reorganized an existing model card into a standard template, while MCGenie generated a new card from repository data.
MCTidy received the original card and a standard template. MCGenie removed existing cards and supplied repository files, generation instructions and a standard template to its language model. After quality filtering and manual curation, the evaluation corpus contained 48 model cards.
Reorganizing existing cards held up better
The MCTidy test began with a checklist of information expected in each card. Checklist preparation produced a median of 69 items, and its average normalized Levenshtein distance from manually corrected checklists was 0.03, a text-difference score used to compare the lists. Against those checklists, MCTidy retained a median of 93.8% of the information. Partial omissions had a median of 1.6%, while complete omissions had a median of 4.5%.
Placement was another relative strength. MCTidy put 98.1% of content in the correct section; 28 of 1,440 sections, or 1.9%, contained confirmed misplaced content.
That strong retention did not remove all errors. The median normalized textual deviation from corrected cards was 0.01, but hallucinations—generated or misinterpreted material—appeared in 147 of 1,440 sections, or 10.2%. The result is a warning about averages: overall textual correction was small by the reported distance measure, while a minority of sections still contained hallucinations.
The outputs were also highly similar across repeated runs. Over three runs, the median full-document semantic similarity was 0.97, and 87.5% of sections reached at least 0.90. Semantic similarity measures how close texts are in meaning; here, it suggests stable output, although it does not by itself show that the result was useful.
Generating new cards exposed the harder problem
MCGenie’s cards looked close to their reference cards at the document level: both mean and median cosine similarity were 0.9. Cosine similarity is another meaning-based comparison, so the score indicates resemblance across a document rather than guaranteeing that every statement is factually correct.
The differences became clearer in sections requiring interpretation. Those sections averaged 0.76 to 0.80 similarity, comparatively lower than the document-level result. The pattern suggests that interpretation-heavy material was harder to reproduce, although section-level differences may also reflect the difficulty and sparsity of the content.
The strongest warning came from the factuality review. An evaluation jury judged 26 of the 48 generated cards, or 54.17%, fully correct. Among the remaining 22 cards, the median number of incorrect subsections was one; the range was one to three, and the mean was 1.41.
The largest reported error category involved generated text that did not accurately reflect the sources it cited. Source-content mismatches accounted for 60% of incorrect information. The result shows why a card can resemble its reference while still failing at the connection between a claim and the file or paper meant to support it.
The researchers also tested what happened when some inputs were left out. Reported document-level similarity was about 0.91 without associated papers, compared with about 0.95 without configuration files and 0.96 without tokenizer files. These comparisons show different results across inputs, but they do not establish that the presence of any one resource causes better output.
The result is a draft, not a guarantee
The authors caution that the 48-card ground-truth dataset was not statistically representative of the broader population. The findings therefore describe performance on a selected evaluation set, not a verdict on model-card repositories more broadly.
The authors released MCTidy and 48 reorganized, manually verified model cards in a replication package.
Taken together, the results make the case for automated model-card systems as drafting and standardization aids. MCTidy was strongest on information retention, section placement and repeatability, while MCGenie often matched reference meaning but still left factual and source-checking gaps.
Paper data and sources
Original title: Automatic Model Card Generation Using an LLM
Authors: Tajkia Rahman Toma, Balreet Grewal, Cor-Paul Bezemer
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text