📄 Arabic-Nougat: Fine-Tuning Vision Transformers for Arabic OCR and Markdown Extraction

Summary

Arabic-Nougat adapts Meta's Nougat, an end-to-end image-to-markup model built for Latin-script scientific PDFs, so that it can read Arabic book pages and output structured Markdown. Arabic is hard for this task because the script is cursive, letter shapes change with position, and page layouts vary. The author fine-tunes three variants (small, base, large) on arabic-img2md, a synthetic set of 13.7k page/Markdown pairs. The pairs were made by scraping Hindawi HTML, rendering it to PDF, and extracting the matching Markdown. The large variant gets a bigger decoder and the Arabic-specific Aranizer-PBE-86k tokenizer, which stretches the effective context to roughly 32k original tokens. Training uses bfloat16 and Flash Attention 2. On 160 held-out pages, the original Nougat models almost completely fail on Arabic, while the fine-tuned models reach low character and word error rates and high structure accuracy. The best model was then used to extract arabic-books, a 1.1-billion-token corpus, which is released together with the models.

Key points

• Architecture: a Donut vision encoder turns a 896×672 page into 588 visual tokens of dimension 1024, and an autoregressive MBart decoder produces the Markdown.
• Model sizes and context: arabic-small-nougat handles up to 2048 tokens, arabic-base-nougat 4096, and arabic-large-nougat 8192. With Aranizer-PBE-86k (86k vocabulary, about 4 Nougat tokens per Aranizer token), the large model's 8192 tokens cover roughly 32k original tokens.
• Results on the 160-page test set:
  • Baseline Nougat models score BLEU 0.0037 and 0.0094.
  • arabic-small-nougat reaches BLEU 0.7565, CER 0.0819 and WER 0.1523.
  • arabic-base-nougat has the lowest WER, 0.1042.
  • arabic-large-nougat has the highest structure accuracy, reported as 98.84% in the results section but 94.7% (with CER 6.1%) in the conclusion. The paper is internally inconsistent here.
• Training setup: learning rate 1e-4, a cap of 100 epochs with early stopping, plus gradient checkpointing and gradient accumulation. Only the base and large models were trained on arabic-img2md; the small model came from earlier training on a smaller dataset.
• Open issues and releases:
  • The small model hallucinates content that is not on the page, such as URLs and images.
  • The larger models loop on long outputs; a repetition penalty helps but does not fix this.
  • The data is biased toward the Hindawi domain.
  • Models, arabic-img2md and arabic-books are all released on Hugging Face.

Interesting ideas

• I1. Gold labels are free when you start from structured source (HTML) and render it into the input modality (page images), because the target structure is known by construction. — For Mova (learn): generate English sentences from hand-built world/frame graphs so their gold UD trees and frames are known in advance, then score parsers on them as a cheap, exact silver-plus check.
• I2. The paper scores layout (Markdown structure accuracy) separately from content errors (CER/WER), so a model can be good at one and weak at the other, as small vs. large shows. — For Mova (back): in the text → graph → text compressor, report structural fidelity (same entities, roles, event order) separately from surface wording, so paraphrases are not punished and lost structure is not hidden.
• I3. A morphology-aware vocabulary cut sequence length by about 4×, which directly increased how much text fits in one decoder window, and the authors track a token-efficiency ratio as a metric. — For Mova (tree): measure how many subword pieces gparse needs per UD token, and check whether a better-fitting vocabulary lets the transformer parser handle longer sentences or documents in a single pass.
• I4. The two failure modes are named and kept apart: content invented that is absent from the input, and degenerate repetition. A decoding-time repetition penalty is treated only as a patch. — For Mova (back): during generation, check that every emitted mention is grounded in a graph node and flag repeated realisations of the same node, giving explainable rejections instead of a tuned penalty.