School-specific filtering was associated with the largest reported gains in a preprint study of Arabic Islamic jurisprudence retrieval. In the study’s main rank-five measure, MRR@5—a score for how high a relevant passage appears among the first five results—the gains on 380 madhhab-based, or Sunni legal-school-specific, questions were 0.315 for Muffakir-fiqh, 0.272 for ATM2-fiqh and 0.260 for BGE-fiqh.
That is a result about finding answer-bearing passages, not about whether a system generates a correct religious ruling. The evaluation stopped at retrieval and did not measure downstream generated-answer quality.
A search benchmark rather than a ruling test
The test drew on approximately 356,000 chunks from 100 classical fiqh books and 503 human-authored questions. The questions included 123 general cases and 380 madhhab-based cases across four Sunni schools.
The benchmark used binary answer-bearing relevance: a passage either qualified as relevant or it did not. It compared dense cosine retrieval, BM25 lexical retrieval and reciprocal-rank fusion, a hybrid that combines the two, with optional madhhab filtering.
The primary scores were reported at the first five results, using MRR@5, nDCG@5 and Hit@5; MAP@5 and Recall@5 were additional measures. The models were tested before and after fiqh-specific fine-tuning, using a separate dataset of 19,319 training triplets drawn from a subset of five fiqh books.
Fine-tuning was linked to higher scores
Among the reported dense baselines, Muffakir had the strongest MRR@5 performance at 0.5238, compared with 0.4407 for ATM2.
Fiqh-specific fine-tuning was associated with higher MRR@5 for all three named models: ATM2 went from 0.4407 to 0.5377, BGE-M3 from 0.4425 to 0.5014, and Muffakir from 0.5238 to 0.5534.
The evidence was uneven across models. The supplied analysis reported significant gains for ATM2 across the metrics, for BGE-M3 on MRR@5 and Hit@5, and non-significant gains for Muffakir.
Hybrid retrieval was slower, with modest score gains
Relative to dense-only retrieval, the reported MRR@5 gains were 0.036 for BGE-fiqh, 0.026 for ATM2-fiqh and 0.021 for Muffakir-fiqh. Mean per-query latency was 2.6, 2.7 and 2.3 times the dense-only latency, respectively.
For ATM2-fiqh and Muffakir-fiqh, the hybrid gains were not statistically significant across the reported metrics.
School-specific questions exposed a retrieval gap
General questions were easier to rank successfully than madhhab-based questions. Fine-tuned ATM2 scored 0.706 on MRR@5 for general questions and 0.483 for school-specific ones; the corresponding Muffakir scores were 0.695 and 0.508.
Recall@5 showed the reverse pattern: 0.075 for general questions and 0.166 for madhhab-based questions. General questions had an average of 41.7 relevant chunks each, compared with 9.7 for madhhab-based questions.
On the 380 school-specific questions, applying madhhab filtering was associated with MRR@5 increases of 0.315 for Muffakir-fiqh, 0.272 for ATM2-fiqh and 0.260 for BGE-fiqh. All reported metric differences were significant at p < 10−24 after Holm correction for multiple comparisons.
Many misses were close, but still missed the answer
Among cases where no answer-bearing passage appeared in the top five, no such passage appeared in the top 20 for 62% of ATM2 misses and 64% of Muffakir misses. The top passage still had a similarity score of 0.618 for ATM2 misses and 0.640 for Muffakir misses.
Manual review identified four recurring error categories: condition mismatch, question-type mismatch, context misinterpretation and topic mismatch. The error analysis focused on Hit@5-zero cases for ATM2 and Muffakir.
The pattern suggests that a passage can be close to a question in subject matter without being the passage that carries the needed answer under the benchmark’s binary relevance rule.
The test stopped short of answer quality
The findings are limited to the supplied Arabic fiqh corpus, its coverage of four Sunni schools and retrieval at rank five. The evaluated models had at most approximately 0.6 billion parameters; larger retrievers and late-interaction models were not tested.
The relevance labels were binary, so they did not capture partial relevance or rulings whose conditions were missing. The lexicon-based madhhab detector was also susceptible to surface-form collisions.
The study does not show that better retrieval improves generated-answer quality, and it does not validate the correctness of generated religious rulings. The authors say the fine-tuning data, trained models and experimental scripts will be released upon acceptance; current availability is not reported.
The document is an arXiv version 1 preprint dated 20 Aug 2026. Its results are therefore benchmark findings within the supplied corpus and school coverage, not a validation of a complete Arabic religious-answering system.
Paper data and sources
Original title: What Makes a Good Fiqh Retriever? Answer Retrieval for Arabic Islamic Jurisprudence
Authors: Somaya Eltanbouly, Heba Sbahi, Samer Rashwani et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text