📄 Enhancing State-of-the-Art NLP Models for Classical Arabic

Summary

Classical Arabic has few annotated datasets and almost no ready-made models, because most Arabic transformers were trained on modern text. The authors study the social history of the Islamic world (c. 600–1600). They build training data and fine-tuned models for biographical collections drawn from the OpenITI corpus. Their starting point is CAMeLBERT-CA, the only Arabic model pretrained on the classical language. They then run an active learning loop. Existing tools pre-annotate biographies: CAMeLBERT-CA-NER, the CAMeL Lab lemmatizer and POS tagger, and the authors' own rule-based date tagger. Domain experts correct this output using a compact markup. The corrected data retrains the models before the next round. The tasks are NER, classifying people by their relation to the biographee, toponym classification, locating the onomastic section, labelling name elements, and detecting and typing dates. Fine-tuning raises person NER F1 from about 60% to about 95%, and the main name elements exceed 97% F1. The models and datasets form a pipeline for building large networks of people, places and dates.

Key points

• The sub-corpus comes from OpenITI and holds 101 multi-volume texts: about 71 million tokens and roughly 495 thousand biographical records. About 70 of these texts date from 1000–1600 CE.

• The NER data grew over two rounds:

  • first round: 1,100 sentences with 3,244 persons, 692 toponyms and 198 MISC entities;
  • second round: 3,826 sentences with 10,333 persons, 1,906 toponyms and 612 MISC entities.

• NER F1 for zero-shot CAMeLBERT-CA-NER versus the fine-tuned model:

  • TOPONYM: 87.11% → 94.42%;
  • PERSON: 60.16% → 94.82%;
  • MISC: 2.84% → 72.68%.

  The authors explain the low zero-shot scores by structure and labelling. The base model learned modern names, which are built differently from traditional ones, and its MISC category barely overlaps with the project's.

• Person relations use six classes, split across two smaller models whose outputs are merged:

  • TEACHER / STUDENT / UNDEFINED;
  • FAMILY / OPINION / CONTACT / UNDEFINED.

• Onomastic element F1 is 98.39% for ISM, 98.57% for KUN, 97.57% for NSB and 97.30% for NAS, but only 77.85% for LQB and 74.73% for SHR. Date-type F1 is strong for BIRTH (93.44%) and DEATH (89.33%) and weak for OFFICE (26.67%), on an evaluation set of 658 date entities.

Interesting ideas

• I1. Bootstrap annotation with an active learning loop: an imperfect pretrained model drafts labels, experts only fix them, and each retrained model drafts the next, larger batch, so expert time goes into correction rather than labelling from scratch.
• I2. Use a markdown-like tag that opens an entity and states its length in tokens instead of a closing tag (e.g. a 3-token teacher-person tag), which makes manual markup fast and less error-prone.
• I3. When a fine-grained label set performs poorly, split it into two smaller classifiers along different axes (here, the learning relation versus the social relation) and merge their outputs afterwards.
• I4. Compare zero-shot and fine-tuned scores per class, because a near-zero class score can expose a mismatch in label definitions rather than a weak model (the MISC case).
• I5. Store vague or conflicting dates at a coarser resolution (decades), and keep both values when a source gives alternative dates instead of forcing a single number.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's silver trees could go through the same draft–correct–retrain rounds, with the LLM teacher or a human fixing witness-parser output and each round's parser seeding the next.
• I2 (learn): A terse, count-based inline notation could speed up hand-correcting spans such as frame arguments or multiword lemmas in silver data, and is easy to parse back into structured labels.
• I3 (frames): Role or speech-act labelling could be factored into independent decisions handled by separate small models, keeping each decision simpler and easier to explain.
• I4 (learn): Per-label baselines against an external model would help tell real parser errors apart from differences between UD dialects or annotation conventions.
• I5 (world): Event timelines could hold interval-valued or multi-valued times, so the world of a text represents uncertainty explicitly rather than inventing precision.