📄 Not Gemma at AR-MS NakbaNLP 2026: Mubsir OCR: End-to-End Recognition of Arabic Handwritten Text

Summary

The paper addresses OCR of historical Arabic handwriting, which is hard because the script is cursive, diacritics are tiny, Eastern and Western numerals appear side by side, and the source pages are often degraded. Instead of a classical pipeline that segments characters, the authors fine-tune end-to-end vision-language models on line crops from the NAKBA shared task (AR-MS, Subtask 2). They run nine controlled experiments, each changing one factor: an external community checkpoint versus in-house fine-tuning, Qwen3-VL-4B-Instruct versus Gemma3, 16 versus 32 epochs, the training prompt, the inference backend (batched vLLM versus sequential HuggingFace), and test-time CLAHE+unsharp enhancement. The best setup is Qwen3-VL-4B, fully fine-tuned for 32 epochs and decoded one image at a time with HuggingFace. It reaches 8.59% CER / 25.87% WER on the annotated val set and 11.00% / 31.26% on the blind test set. The study shows that in-domain fine-tuning matters most, while prompt changes and image preprocessing contribute little.

Key points

• Data: 15,962 training line crops and 2,095 annotated val lines. A separate blind AR-MS set of 2,671 images can only be scored by official submission. The paper says in-house models were trained on merged train+val splits, which may affect how far the val scores can be trusted.
• The external Qwen2.5-VL-3B checkpoint (Sherif, 2025b) scores 20.20% CER / 51.52% WER. NAKBA fine-tuning of Qwen3-VL-4B brings this down to 12.34% CER after 16 epochs and about 10.5–11.4% after 32 epochs.
• With the same checkpoint and the same greedy decoding, the backend alone changes the result. vLLM gives 10.49% CER and HuggingFace sequential inference gives 8.59%, but HuggingFace is about 8× slower (≈3.3 vs ≈0.4 s/image). The authors blame precision and padding effects, although both runs are reported as bf16.
• Changing the training prompt moves CER by only 0.26 pp. Gemma3 reaches 11.10% CER / 29.80% WER and runs about 2.6× slower than Qwen3-VL-4B.
• Preprocessing results:
  • Aggressive CLAHE (8×8 tiles) made 848 lines worse and 466 better, a net loss.
  • A gentler v2 with a skip rule for small images lowered CER from 11.15 to 10.18%, with almost no WER change.

Interesting ideas

• I1. The study is built from controlled pairs: two runs share everything except one factor, such as prompt, backend, or preprocessing, so each effect can be attributed cleanly.
• I2. Instead of reporting only an averaged score, the authors count how many items each change improved and how many it hurt (848 vs 466), which exposes a harmful change even when the mean looks fine.
• I3. Applying a transformation only at test time, to inputs the model never saw during training, creates train–test mismatch and gives unreliable gains.
• I4. The same trained model gave different accuracy depending on whether inputs were batched with padding or processed one at a time, so the serving path itself needs testing.
• I5. The ratio between character and word error rates (about 1:3) is read as a diagnostic, and the remaining errors are sorted into named classes: diacritics, numeral script, and ligature split/merge.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation of repair rules or teacher tiers could always run as such matched pairs plus a negative control, so a reported gain is tied to exactly one change.
• I2 (learn): Self-induced repair rules could be accepted only if the number of sentences they fix clearly outweighs the number they break in the silver data.
• I3 (tree): Any input normalization or UD dialect conversion used at inference time in gparse should also be applied to its training treebanks, or it should be left out.
• I4 (tree): Add a check that gparse produces identical trees for a sentence parsed alone and inside a padded batch.
• I5 (tokens): Pair token-level error rates with word- and sentence-level ones, and keep a named error taxonomy, to locate where the tagger's or lemmatizer's mistakes cascade.