📄 RefLAM: A Reference-Grounded Line Annotation Pipeline for Historical Arabic Manuscripts

Summary

Historical Arabic manuscripts are photographed in large numbers, but few have line-level transcriptions that could train handwritten-text-recognition (HTR) models. Annotating by hand is slow, at about 40 lines per hour. RefLAM speeds this up by combining two existing resources: a multimodal LLM that does zero-shot, layout-tagged OCR of whole pages, and a clean scholarly edition of the same text that is not tied to page images. The pipeline segments lines with a Muharaf-trained Kraken model, runs Gemini OCR, normalises both texts in the same diacritic-insensitive way, finds where each page sits in the book-level reference, and aligns each OCR line to a contiguous reference span with a confidence score from 0 to 100. The authors prove that a score of 100 means the normalised strings are identical. Reviewers can therefore confirm such lines almost instantly and spend their effort on uncertain ones. The result is a 75× throughput gain and the released corpus AraMS-28k. HATFormer and Kraken baselines finetuned on it show the corpus is usable for HTR training.

Key points

• Throughput on 7 fully page-validated books rose from 40 to 3,000 lines/hr (75×). For 7 further books, 16,533 confidence-100 main-text lines were kept within one week, and lower-scoring lines were dropped rather than corrected.
• AraMS-28k contains 14 books and 3,043 pages, with 27,971 main-text and 629 margin-line annotations. Of the margin entries, 191 (30.4%) carry insertion anchors. It covers Naskh, Ruq'ah and Maghrebi hands plus one lithographed volume.
• The tools are a Kraken segmentation model trained on Muharaf, with polygons converted to boxes for easier correction, and Gemini (gemini-3-flash-preview) driven by a versioned prompt (v2.3).
• The normalisation step:
  • strips harakat, superscript alef and kashida;
  • unifies alef variants, alef-maqsura and ta-marbuta;
  • removes non-Arabic symbols. Both raw texts are kept (gt_raw, gemini_raw).
• A manual audit found no wrong line among those scoring 100. The main limitation is that roughly 70% of margin lines could not be anchored confidently, and a clean transcription must already exist.

Interesting ideas

• I1. A confidence metric whose top value is formally the same as exact equality turns review from uniform checking into triage, because a perfect score can be trusted without inspection. — For Mova (learn): Silver data from witness parsers could auto-accept only arcs where a provably exact agreement criterion holds, and send everything else to LLM teacher tiers or human checks.
• I2. Treating fluent LLM output strictly as a noisy hypothesis, and checking it against a trustworthy but unaligned source, makes the alignment score work as a hallucination detector at no extra cost. — For Mova (learn): LLM teacher annotations (frames, roles, answers) could be scored against an independent reference such as the source sentence tokens or VerbNet constraints, with low-agreement items quarantined instead of trained on.
• I3. Applying one canonical normalisation operator identically to both sides before comparison, while storing both raw strings, keeps matching robust without losing information for later tasks. — For Mova (tokens): A documented normaliser for quotes, apostrophes, hyphens and case could feed lemma and token matching, with raw forms retained so that generation and error analysis still see the original text.
• I4. The authors released only lines that met the strict guarantee and excluded uncertain ones rather than hand-fixing them, which keeps the released data's precision claim honest. — For Mova (tree): Treebank conversion between UD dialects could ship only sentences whose round-trip conversion reproduces the input exactly, and list the rest separately for repair rules.
• I5. Keeping a version history of the LLM prompt, with what each revision fixed, makes the pipeline auditable and reproducible together with cached model responses. — For Mova (learn): Teacher-tier prompts and their cached outputs could be versioned the same way, so every silver label can be traced to the exact prompt that produced it.