📄 AlSaifTeam at AR-MS NAKBA-NLP 2026: Building Expert-Quality Ground Truth for Arabic Handwritten Manuscripts

Summary

This paper describes a team's entry to the first subtask (Subtask 1) of the Arabic Manuscript Understanding Shared Task held at NAKBA NLP 2026. The subtask asks for trustworthy line-level ground truth for handwritten Arabic, here pages of the Omar Al-Saleh Memoir. The authors do not train a recognizer. Instead, they focus on how people should annotate:

• Transcribers follow fixed orthographic conventions designed for historical script.
• Other team members check each line independently, and a double-blind verification stage is added.
• Every annotation then passes a confidence gate with three outcomes: keep it, send it back for more paleographic context or a supervisor's review, or drop it as too ambiguous.

Historical lexical dictionaries and an interface that shows the image beside the typed text support the work. Across 150 high-resolution pages, the team reports a Character Error Rate 12% lower than single-pass annotation and an estimated final accuracy of 98.4%. The short paper gives little detail on how these numbers were obtained. Its main contribution is a repeatable annotation recipe for building benchmark data in low-resource Arabic HTR.

Key points

• The paper addresses Subtask 1 of the NAKBA NLP 2026 shared task: manual line-level transcription of the Omar Al-Saleh Memoir.
• The confidence gate has three bands: accept above 90% (full annotator agreement), re-evaluate between 60% and 90%, and discard below 60%.
• The setup covers 150 high-resolution pages, a custom side-by-side transcription tool, and historical lexical dictionaries for cursive ambiguities.
• The authors report that multi-stage verification cut CER by 12% compared with single-pass manual annotation.
• They estimate final ground-truth accuracy at 98.4%, which they credit mainly to better handling of difficult historical ligatures.

Interesting ideas

• I1. A three-way gate (accept, escalate, discard) is used instead of a binary keep/drop, so mid-confidence items get extra attention rather than being lost or blindly trusted.
• I2. Ambiguous items are deliberately removed from the benchmark itself, keeping the gold set free of disputed answers at some cost to coverage.
• I3. Fixed written transcription rules are agreed before any annotation starts, which reduces drift between annotators.
• I4. The value of the verification stage is measured by comparing error against a single-pass baseline, rather than simply assuming extra review helps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Arcs on which all witness parsers agree become silver data, split arcs go to an LLM teacher tier, and heavily contested arcs are never trained on.
• I2 (answer): Test sets for school-course and fable QA could drop questions where human or teacher answers disagree, so failures reflect real reader errors rather than noisy gold.
• I3 (tokens): A short, versioned convention sheet for contractions, hyphenation and multiword lemmas would keep tokenizer output and hand-checked samples consistent across contributors.
• I4 (learn): Mova could check, against a small gold sample, whether each additional witness parser or repair-rule pass actually lowers error before it becomes part of the pipeline.