Elamur Research⌕
☰
home / papers / 2026 / Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?

Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?

Summary

This work asks whether today's LLMs can write maqamat, the classical Arabic prose genre built on rhymed prose (saj'), heavy rhetorical ornament and episodic stories about a witty trickster. Arabic generation research has mostly covered poetry, MSA and dialects, so this is the first controlled study of the genre. The authors prompt five models (GPT-4o, GPT-5.4-mini, ALLaM-7B-Instruct, Qwen3-8B, LLaMA-3-8B-Instruct) on 14 classical-style themes. Each model is run in three regimes: zero-shot, few-shot with real excerpts, and rule-based instructions. They then grade the outputs in four ways:

  • a Claude Sonnet judge using absolute rubric scores,
  • the same judge making pairwise comparisons,
  • two expert human annotators,
  • a second LLM judge and lightweight non-LLM saj' proxies.

GPT-5.4-mini leads, GPT-4o follows, and the open-weight models trail far behind. Saj' responds most strongly to prompting, especially few-shot. Creativity and rhetoric change little, which suggests deeper literary skill depends on the model rather than the prompt. The study matters because fluent Arabic clearly does not guarantee mastery of a tradition-bound style. The authors release the topics, prompts and outputs.

Key points

  • The generation grid covers 5 models × 14 topics × 3 prompting strategies × 5 samples per cell, run at temperature 0.7 with up to 2400 tokens. All outputs are stored as JSONL with metadata.
  • The judge is claude-sonnet-4-5-20250929, called through structured tool use at temperature 0. It rates each output on a 1–5 Likert scale across eight criteria: overall quality, authenticity, topical relevance, creativity, coherence, richness of rhetoric, saj' and grammatical correctness. It also judges pairwise with randomized order, over a stratified sample of 420 pairs.
  • For human evaluation, two native-speaker experts made blind forced-choice judgments on 60 maqamat (30 pairs). The pairs covered all ten model pairings and also compared the same model under different prompts.
  • Few-shot prompting improved saj' most reliably, and zero-shot gave the best aggregate scores across all five models. The strongest models, GPT-4o and GPT-5.4-mini, gained most from rule-based prompts on rhetoric and coherence. ALLaM sometimes got worse when prompted.
  • Simple non-LLM saj' proxies correlated with the judge's saj' scores (Spearman's ρ = 0.328, p < 0.001). Few-shot prompts sharply raised diacritic density, for example Qwen rose from 30.6 to 77.3 marks per 100 letters.

Interesting ideas

  • I1. The authors check that absolute rubric scores and head-to-head comparisons give the same model ranking, which makes it unlikely the ranking comes from one scoring format. — For Mova (learn): When an LLM teacher tier grades silver parses or frame outputs, run both a per-item score and a pairwise preference. Only trust tiers whose rankings agree under both formats.
  • I2. A crude, transparent measurable proxy is correlated with the LLM judge's scores to show the judge tracks something real rather than arbitrary taste. — For Mova (learn): Pair each LLM-judged dimension with a deterministic signal, such as agreement between witness parsers or frame-slot completeness. Use the correlation as a sanity check on the teacher.
  • I3. In pairwise judging, candidate positions are randomly swapped before judging and mapped back afterwards, which neutralizes the judge's bias toward first or second position. — For Mova (learn): Apply the same swap-and-remap step whenever an LLM chooses between two candidate trees or two repair rules. This keeps position bias out of the training signal.
  • I4. Diacritic density jumped only with few-shot prompts, which shows models copy surface features of the in-context examples rather than having the behavior themselves. — For Mova (learn): Treat exemplar-copying as a negative control when using few-shot LLM teachers. Check whether gains are just imitated formatting or lexical habits from the shown examples.
  • I5. Outputs are judged on separate dimensions (grammar, coherence, relevance, style, and others), which reveals that some qualities respond to prompting while others stay fixed. — For Mova (back): Score text regenerated from the meaning graph per dimension rather than with one number. This would separate fixable surface problems from real gaps in the graph → text path.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.28245 · arXiv 2609.28245

@misc{morsy2026beyond,
  title = {{Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?}},
  author = {Morsy, AbdulRahman A. and Science, Aya Zirikly   Department of Computer and Engineering, School of and Sciences, Applied and University, George Washington and DC, Washington and States, United and Speech, Center for and Processing, Language and Engineering, Whiting School of and University, Johns Hopkins and MD, Baltimore and States), United},
  year = {2026},
  doi = {10.48550/arxiv.2609.28245},
  eprint = {2609.28245},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.28245}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wafa Al Ghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan Pravin More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, and Rao Muhammad Anwer. 2025. Fann or flop: A multigenre, multiera benchmark for arabic poetry understanding in llms. find
  2. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. find
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. find our card
  4. Sina Bagheri Nezhad. 2026. Toward culturally grounded natural language processing. arXiv preprint arXiv:2603.26013. ↗
  5. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗ our card
  6. Basma El Amel Boussaha, Mohammed Alyafeai, Ahmed Alzubaidi, Leen Al Qadi, Shaikha Alsuwaidi, and Hakim Hacid. 2025. Adapting falcon3-7b language model for arabic: Methods, challenges, and outcomes. In Proceedings of The Third Arabic Natural Language Processing Conference. find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. find
  8. Rina Drory. 2000. The maqama. In María Rosa Meno-cal, Raymond Scheindlin, and Michael Sells, editors, The Literature of Al-Andalus, pages 190–210. Cambridge University Press. find
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898. find
  10. Jaakko Hämeen-Anttila. 2002. Maqama: A History of a Genre, volume 5 of Diskurse der Arabistik. Harrassowitz Verlag. find
  11. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104. find our card
  12. Danielle Lewis, Andrea Zugarini, and Eduardo Alonso. 2021. Syllable neural language models for english poem generation. In Proceedings of the 12th International Conference on Computational Creativity (ICCC), pages 350–356. find
  13. Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? In Proceedings of the 15th International Conference on Computational Creativity (ICCC). find
  14. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline-conditioned generation with dynamic plot state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4274–4295. find
  15. Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, and Fajri Koto. 2026. Instruction-guided poetry generation in arabic and its dialects. arXiv preprint arXiv:2604.27766. ↗
  16. Neha Sengupta et al. 2023. Jais: Arabic-centric foundation and instruction-tuned large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. find our card
  19. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/morsy-2026-poetry-large-language-models-generate/