Elamur Research⌕
☰
home / papers / 2026 / AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task

AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task

Paper · 2026 · Mo El-Haj, Saad Ezzini, Shadi Abudalfa et al.

Summary

Arabic genre identification is hampered by scarce labelled data and by wide variation across Modern Standard Arabic, Classical Arabic, dialects, diacritisation and spelling habits. AraGenre frames the problem as hierarchical label generalisation. Each text segment receives one of six broad communicative genres plus a fine-grained genre. At test time, the fine-grained inventory consists of 74 genres that never appeared with labels and are known only through English names and definitions. The organisers drafted these definitions with ChatGPT 5 and then corrected them by hand. Training and development data are tiny and mostly synthetic. The hidden test set holds nearly 28k natural segments drawn from ArabicNLP.uk corpora such as Tarab, AraFinNews, ArabJobs, HABIBI and KALIMAT. Seventeen teams completed the task, and Thaka led with a Hierarchical Macro F1 of 0.7352. Every team beat the embedding-similarity baseline (0.2472). Broad genres were recognised far more reliably than specific ones, and scores dropped sharply from development to test. This shows that near-perfect results on clean, controlled data reveal little about robustness to unseen labels and noisy real-world text.

Key points

  • Data splits: training has 140 examples (5 broad, 7 specific genres), development has 110 (6 broad, 6 specific), and the final test has 27,972 natural segments with 74 unseen specific genres. Of these, 42 are Informative, 12 Learning, 7 Creative, 6 Religious, 4 Interactive and 3 Legal.
  • Ranking metric: Hierarchical Macro F1 is the equal-weight mean of broad and specific Macro F1. The two levels are scored independently, with no hierarchy-consistency check. Evaluation ran on Codabench using scikit-learn.
  • Participation and top results: 46 registrations, 373 submissions and 17 final teams. The top three were Thaka (0.7352), HP (0.7169) and NAMAA (0.7013). Thaka won on specific-genre F1 even though HP was slightly ahead on broad genres.
  • Broad vs specific gap: across teams, mean broad Macro F1 was 0.6509 against 0.3238 for specific genres, a gap of 0.3271.
  • Dev-to-final drop: among the 13 teams whose final entries could be linked to their development runs, the average score declined from 0.9344 to 0.5261. Baseline 1, a multilingual sentence encoder matched against definitions, scored 0.4658 on development versus 0.3899 for the RAG baseline, and dropped to 0.2472 on the final test.

Interesting ideas

  • I1. Classes are represented by natural-language definitions rather than fixed IDs, so new labels can be added at test time without retraining and each decision can be traced back to the definition it matched.
  • I2. Development data is deliberately kept clean and synthetic while the test set is natural and noisy, and the dev-to-final drop is reported as a finding in its own right rather than treated as noise.
  • I3. Scoring coarse and fine labels separately, then averaging them, exposes systems that get the family right but the subtype wrong.
  • I4. The scorer counts missing items as an INVALID label and keeps only the last prediction for duplicated IDs, which removes any benefit from skipping hard items or hedging with several answers.
  • I5. Label definitions were first drafted by an LLM, then hand-edited so that each one clearly separates its genre from neighbouring categories.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): when a predicate lacks a VerbNet entry, candidate frames could be ranked by matching the clause against FrameNet/VerbNet definitions, with the winning definition kept as an explanation.
  • I2 (learn): models trained on silver or synthetic data should always be scored on a separate set of naturally occurring text, with the size of the gap tracked as a standing robustness metric.
  • I3 (tree): parser evaluation could report LAS on universal relations and on subtyped relations (e.g. nsubj vs nsubj:pass) side by side to locate where errors concentrate.
  • I4 (answer): test harnesses for the school-test and fable-QA readers could treat abstentions as errors and accept exactly one answer per question, so accuracy cannot be inflated.
  • I5 (world): an LLM teacher could draft glosses for verb classes or ontology nodes, with human review focused specifically on how each one differs from its nearest siblings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia52 works · 4,666 citations🎓Lancaster UniversityLancaster · United Kingdom35 works · 1,615 citations🎓VinUniversityHanoi · Vietnam11 works · 56 citations🎓University College of Applied ScienceGaza · Palestine3 works · 2 citations

Also on the byline: Onaizah Colleges.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mo El-Haj, Saad Ezzini, Shadi Abudalfa, Mustafa Jarrar, Nguyen Minh Chi and Nguyen Minh Quan. 2026. AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task. arXiv:2609.27387 · The 4th ArabicNLP conference at EMNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.27387 · arXiv 2609.27387

@misc{elhaj2026aragenre,
  title = {{AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task}},
  author = {Mo El-Haj, Saad Ezzini, Shadi Abudalfa, Mustafa Jarrar, Nguyen Minh Chi and Nguyen Minh Quan},
  year = {2026},
  doi = {10.48550/arxiv.2609.27387},
  eprint = {2609.27387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.27387}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rabee Al-Qasem. 2026. Rabee Al-Qasem at AraGenre 2026: Conformer based hierarchical classifier for arabic genres. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  2. Hassan Alqaeri, Meshal Alamr, and Abdullah Aldahlawi. 2026. Thakaa at AraGenre 2026: Definition-guided LLM judging with full-taxonomy agreement-gated correction for hierarchical Arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  3. Manar Ali Alsubhi and Ghader Reda Kurdi. 2026. GHADAI at AraGenre 2026: Definition-guided classification of arabic text genres via bi-encoder retrieval and large language model reranking. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  4. Shahad Altamimi, Samar Ahmed, Morooj Alrajhi, Yahya Assiri, and Abdullah I. Alharbi. 2026. Nasaq at AraGenre 2026: Definition-guided approaches for hierarchical arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  5. Hassan Barmandah, Israa Elhosiny, Yousra El-Ghawi, and Omer Nacar. 2026. NAMAA community at AraGenre 2026: From encoder baselines to self-consistent LLM ensembling for hierarchical arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  6. Najat Bostaty, Salam Almasri, Bushra Al Shaa-bani, Raghad Al Hossny, and Riad Sonbol. 2026. WathiqTeam at AraGenre 2026: Class-balanced hybrid retrieval with selective LLM reranking. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  7. Nguyen Minh Chi. 2026. Phoenix at AraGenre 2026: Transductive bias correction and multi-view label representations for zero-shot arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  8. Chiamaka Ijeoma Chukwuneke, Paul Rayson, Ignatius Ezeani, Mo El-Haj, Doris Chinedu Asogwa, Chidimma Lilian Okpalla, and Chinedu Emmanuel Mbonu. 2023. Igboner 2.0: Expanding named entity recognition datasets via projection. In 4th workshop on african natural language processing. find
  9. Hieu Pham Dinh, Hung Nguyen Huy, and Mo El-Haj. 2026. Vietjobs: A vietnamese job advertisement dataset. arXiv preprint arXiv:2603.05262. ↗
  10. Mahmoud El-Haj. 2020. Habibi-a multi dialect multi national arabic song lyrics corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1318–1326.
  11. Mahmoud El-Haj and Rim Koulali. 2013. Kalimat a multipurpose arabic corpus. Culture, 2:1–359. find
  12. Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox. 2010. Using mechanical turk to create a corpus of arabic summaries. In European Language Resources Association (LREC 2010). European Language Resources Association (ELRA). find
  13. Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox. 2015. Creating language resources for under-resourced languages: methodologies, and experiments with arabic. Language Resources and Evaluation, 49(3):549–580. ↗
  14. Mahmoud El-Haj, Paul Rayson, and Mariam Aboelezz. 2018. Arabic dialect identification in the context of bivalency and code-switching. In Proceedings of the 11th International Conference on Language Resources and Evaluation, Miyazaki, Japan., pages 3622–3627. European Language Resources Association.
  15. Mo El-Haj. 2025. Arabjobs: A multinational corpus of arabic job ads. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 16–25. ↗
  16. Mo El-Haj. 2026. Tarab: A multi-dialect corpus of arabic lyrics and poetry. In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script, pages 37–46. ↗
  17. Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, and Ruslan Mitkov. 2024. DARES: Dataset for Arabic readability estimation of school materials. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 103–113, Torino, Italia. ELRA and ICCL.
  18. Mo El-Haj and Saad Ezzini. 2024. The multilingual corpus of world’s constitutions (mcwc). In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024, pages 57–66. find
  19. Mo El-Haj and Paul Rayson. 2025. Arafinnews: Arabic financial summarisation with domain-adapted llms. In 2025 IEEE International Conference on Big Data (BigData), pages 5428–5436. IEEE.
  20. Alaa M El-Halees. 2017. Arabic text genre classification. Journal of Engineering Research & Technology, 4(3):105. find
  21. Ashraf Elnagar, Ridhwan Al-Debsi, and Omar Einea. 2020. Arabic text classification using deep learning models. Information Processing & Management, 57(1):102121. ↗
  22. Ignatius Ezeani, Mahmoud El-Haj, Jonathan Morris, and Dawn Knight. 2022. Introducing the welsh text summarisation dataset and baseline systems. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5097–5106. find
  23. Zeyd Ferhat, Abir Betka, Riyadh Barka, Zineddine Kahhoul, Selma Boutiba, Mohamed Tiar, Habiba Dahmani, and Ahmed Abdelali. 2024. Functional text dimensions for arabic text classification. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 352–360.
  24. Lingyu Gao, Debanjan Ghosh, and Kevin Gimpel. 2023. The benefits of label-description training for zero-shot text classification. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 13823–13844. find
  25. Hung Nguyen Huy, Mo El-Haj, Dawn Knight, and Paul Rayson. 2026. Freetxt-vi: A benchmarked vietnamese-english toolkit for segmentation, sentiment, and summarisation. arXiv preprint arXiv:2603.05690. ↗
  26. Nouran Khallaf, Elin Arfon, Mo El-Haj, Jonathan Morris, Dawn Knight, Paul Rayson, Tymaa Hasanain Hammouda, and Mustafa Jarrar. 2023. Open-source thesaurus development for under-resourced languages: a welsh case study. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 306–315. find
  27. Dawn Knight, Nouran Khallaf, Paul Rayson, Mahmoud El-Haj, Ignatius Ezeani, and Steve Morris. 2024. Freetxt: A corpus-based bilingual free-text survey and questionnaire data analysis toolkit. Applied Corpus Linguistics, 4(3):100103. ↗
  28. Daisy Monika Lal, Paul Rayson, and Mo El-Haj. 2025. Hindi reading comprehension: Do large language models exhibit semantic understanding? In Proceedings of the First Workshop on Natural Language Processing for Indo-Aryan and Dravidian Languages, pages 1–10. find
  29. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.
  30. Lei Li, Corina Forascu, Mahmoud El-Haj, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 1: Arabic, English, Greek, Chinese, Romanian. In Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 1–12, Sofia, Bulgaria. Association for Computational Linguistics. find
  31. Yasmina Mansour, Ahmad Afles, and Mohamad Ballout. 2026. SAGE at AraGenre 2026: Adapter-refined definition embeddings for zero-shot arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  32. MohannadHendi. 2026. Mohannadhendi at AraGenre 2026: Contrastive cross-lingual definition retrieval for hierarchical arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  33. Lorenzo Paletto, Valerio Basile, and Roberto Esposito. 2024. Label augmentation for zero-shot hierarchical text classification. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7697–7706. find
  34. Dinh Hoang Phong and Dang Van Thin. 2026. HP at AraGenre 2026: LLM-guided pseudo-labeling for hierarchical arabic genre classification with MARBERT. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  35. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992. ↗
  36. Eman Sedqy Shlkamy and Wajdi Zaghouani. 2026. Northwestern at AraGenre 2026: Definition-guided dense retrieval using BGE-M3 for hierarchical arabic genre classification. In Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026), Budapest, Hungary. Association for Computational Linguistics. find
  37. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/el-haj-2026-aragenre-2026-hierarchical-definition-guided/