Instruction-fine-tuned language models recorded higher scores than prompting-only systems in a benchmark designed to recover meaningful Sanskrit phrases and produce meanings grounded in a supplied English translation. The work is an arXiv version 1 preprint dated 25 August 2026.
Phi-4 had the strongest overall reported performance. With instruction fine-tuning, it recorded a Jaccard score of 0.716, compared with 0.092 without examples, 0.358 with one example, 0.230 with five examples and 0.299 with 10 examples. Jaccard measures overlap between the glossary phrases a model generates and the reference phrases.
The same phi-4 comparison showed a Meaning Faithfulness score of 0.787 with fine-tuning, against 0.182 zero-shot, 0.635 with one example, 0.604 with five and 0.628 with 10. Meaning Faithfulness combines key matching with similarity between generated and reference meanings. The prompted conditions therefore did not improve in a steady progression as more examples were added, and all remained below the fine-tuned result in the reported comparison.
A focused test of Sanskrit phrase meanings
The benchmark contained 31,316 Sanskrit sloka, translation and glossary triples drawn from the Valmiki Ramayana and Srimad Bhagavatam. The data were divided into 25,050 training examples, 3,133 validation examples and 3,133 test examples. Contiguous verses and near-duplicate glossary entries were kept from crossing between the splits.
The study compared four named language models: gemma-3n-E4B-it, gemma-3-12b-it, phi-4 and Qwen3.5-9B. It tested prompting with no examples and with one, five or 10 examples, alongside instruction fine-tuning. FastAlign and ByT5-Sanskrit were included as baselines.
Meaning Faithfulness matched generated and reference keys using normalized Levenshtein similarity with a threshold above 70, then compared their meanings through cosine similarity between SentenceTransformer embeddings.
Phrase boundaries changed the picture
A separate segmentation comparison produced a sharp contrast. Phi-4 paired with gold segmentations recorded a perfect Jaccard score of 1.0 and a Meaning Faithfulness score of 0.872. When the system used segments predicted by ByT5-Sanskrit, the corresponding scores were 0.501 and 0.741.
The figures describe a difference between two test conditions; they do not by themselves establish that segmentation caused the difference.
The lowest-scoring cases often split compounds
The researchers also reviewed 174 samples from the bottom 5% by Meaning Faithfulness. Over-segmentation was the dominant categorized error, appearing in 122 cases, described in the analysis as 70% of the sample. The failures included fragmentation of sandhi and samasa compounds.
That review was limited to the lowest-scoring cases, so it may not represent the full distribution of errors. The authors identify morphology and compound-boundary detection as primary bottlenecks in the task.
In the reported seen and unseen subsets, phi-4 had 2,626 correct seen-key predictions, or 21.53%, in zero-shot prompting, compared with 10,713, or 87.85%, with instruction fine-tuning. For unseen keys, the corresponding figures were 564, or 6.30%, and 5,952, or 66.45%. The comparison associates instruction fine-tuning with higher correct-key rates for both groups.
A word-order robustness check randomly shuffled words within each sloka line while preserving line boundaries. Under that perturbation, Qwen3.5-9B recorded a Jaccard score of 0.394 and a Meaning Faithfulness score of 0.682, with the analysis describing noticeable degradation, particularly for that model.
What the benchmark cannot answer
The evidence is limited to examples from the Valmiki Ramayana and Srimad Bhagavatam. That leaves generalization to other Sanskrit literary traditions demonstrated only to a limited extent.
The scores also have an important measurement caveat. Jaccard and Meaning Faithfulness rely on surface string matching and embedding similarity, and they may penalize valid sandhi resolutions that use a different orthographic form. A lower score can therefore reflect a written-form difference as well as a mismatch in the recovered phrase or meaning.
Instruction fine-tuning used memory-constrained LoRA settings, while the qualitative error analysis covered only the bottom 5% by Meaning Faithfulness. The supplied analysis reports no confidence intervals or inferential uncertainty estimates for the comparisons.
The findings should therefore be read as performance results within this benchmark and its tested conditions. They do not establish glossary correctness beyond the benchmark distribution.
The abstract states that the data and code are available. The work was supported in part by the GCP Research Grant for Gemma and the Government of India NLTM Bhashini project.
Paper data and sources
Original title: Padamitra: Grounded Glossary Generation for Classical Sanskrit
Authors: Manoj Balaji Jagadeeshan, Sai Pragnaan Marala, Pawan Goyal
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text