📄 Generating Classical Arabic Poetry using Pre-trained Models

Summary

The paper asks whether general pre-trained transformers can absorb the formal rules of classical Arabic verse (meter, i.e. al-'arūd, and rhyme, al-qafiya) from data alone. Earlier systems instead built these constraints in by hand. The authors merged the ashaar and Arabic Poetry Dataset collections and scraped al-diwan to fill gaps and add a rhyme field. They kept only poems from pre-modern eras with strict prosody and restricted the corpus to 15 of the 16 canonical meters. On this data they fine-tuned two architectures:

• a BERTShared encoder-decoder, warm-started from CAMelBERT-CA;
• the 6B-parameter decoder-only GPT-J.

Each model was conditioned once on a meter prompt and once on a topic prompt. An automatic check measured meter adherence, and expert human raters scored rhyme consistency, fluency and coherence. Raters found BERTShared output somewhat more fluent and coherent, while GPT-J kept rhyme noticeably better. The work suggests that off-the-shelf models capture some poetic structure without explicit constraint machinery.

Key points

• Corpus: ashaar (HuggingFace), the Kaggle Arabic Poetry Dataset and al-diwan scraping, totalling about 18K poems.
  • Eras include Abbasid, Umayyad, Andalusian, Mamluk, Ottoman, Ayubi and Pre-Islamic.
  • Labels cover 17 topics, with al-T.awīl, al-Kāmil, al-Basīt. and al-Wāfir the dominant meters.
• BERTShared design: encoder and decoder share parameters, which halves memory.
  • The WordPiece tokenizer was extended with two tokens, one splitting hemistichs and one marking verse starts.
  • Poems were chunked into 23-verse pieces, with a 512-token output limit.
• BERTShared training and decoding: batch size 16, Adam at 5e-5 with 3k warm-up steps, on a T4 GPU in Colab.
  • Generation used beam search with multinomial sampling, capped at 130 tokens.
  • The meter experiment used a 15,000-poem sample with an 85/15 split.
• GPT-J: decoded with top-p sampling and trained on TPUs from Google TRC. It was only fine-tuned, not pre-trained, which the authors call suboptimal.
• Data cleanup and limitations:
  • Poems rhyming on a short vowel were relabelled with the matching long vowel.
  • The catch-all topics "generic" and "short" were dropped for topic prompting.
  • The authors acknowledge unverifiable topic labels and evaluator fatigue across many meters.

Interesting ideas

• I1. Encoding the verse layout with dedicated separator tokens lets a generic model learn hemistich and verse boundaries without architectural changes.
• I2. Collapsing phonetically identical variants (short vowel vs. long vowel rhyme) into one canonical label prevents spurious class splits in the data.
• I3. Removing uninformative catch-all labels ("generic", "short") keeps the conditioning signal meaningful, even though it costs data.
• I4. Evaluation is split between an automatic test for the hard formal constraint (meter) and human judgement for soft qualities (fluency, coherence).
• I5. Warm-starting both halves of an encoder-decoder from one domain-matched checkpoint with shared weights gives a cheap seq2seq model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): when realizing text from the meaning graph, explicit boundary tokens for clauses and sentences could make the structure of generated output controllable and checkable.
• I2 (tokens): mapping spelling variants such as colour/color to a canonical form before lemmatization would keep morphology tables and counts consistent.
• I3 (learn): when building silver data, examples whose label is a vague default could be excluded so that witness agreement reflects real distinctions.
• I4 (back): the text → graph → text compressor could be scored by an automatic structural round-trip check, with human or LLM review kept only for naturalness.
• I5 (back): a translation or regeneration model through the meta-world could reuse one shared encoder checkpoint to fit modest GPU memory.