Elamur Research⌕
☰
home / papers / 2024 / DRU at WojoodNER 2024: ICL LLM for Arabic NER

DRU at WojoodNER 2024: ICL LLM for Arabic NER

Paper · 2024 · Nancy Hamdan, Hadi Hamoud, Chadi Abou Chakra et al.

Summary

The paper describes a system for the open track of the WojoodNER 2024 shared task, specifically flat fine-grained Arabic NER (Subtask 3A). The data comes from five news domains covering the war on Gaza. Instead of training a tagger, the authors used in-context learning with Cohere's Command R, run in its 8-bit quantized form. They wrote a long system prompt derived from the Wojood-Fine annotation guidelines and added 27 hand-picked demonstrations. The prompt asks the model to emit a CSV row per token containing a top-level tag and a subtype tag in BIO notation. The model's output was often malformed: tokens were missing or extra, some labels did not exist in the tag set, and some rows broke the format. To handle this, the authors built a rule-based repair step. The main configuration reached F1 0.737. Repair raised recall slightly without improving F1. Splitting the task across several smaller prompts performed clearly worse. The work matters because it shows that a mid-sized open LLM can do reasonable Arabic NER from guidelines alone. It also documents the practical cost of keeping such a model's output consistent.

Key points

  • Subtask 3 data: train has 1,928 sentences (50,144 tokens) and dev has 372 sentences (10,049 tokens), with flat and nested versions annotated under the Wojood-Fine guidelines.
  • Model: Command R (35B parameters, context up to 128K tokens), which ranks second on the Open Arabic LLM Leaderboard. It was used 8-bit quantized.
  • Scores on dev:
    • Strict-format output: P 0.719, R 0.756, F1 0.737.
    • After repair: R 0.759, F1 0.735.
    • Multi-level split prompts: F1 0.637.
    • Organizers' evaluation without the "O" tag: much lower, F1 about 0.29–0.31 for the main runs.
  • Observed output defects:
    • Only 9,915 of 10,049 tokens were returned.
    • The model produced 14 invented tags.
    • Only 9,416 token rows followed the required format exactly, and six recurring malformation patterns were identified.
  • Tags with the strongest results were LAW, GPE, PERS, ORG and WEBSITE. EVENT and QUANTITY were near zero.

Interesting ideas

  • I1. The three-level hierarchy is collapsed into two output columns: a compound subtype such as GPE_ORG is expressed as two co-occurring top-level tags, and the compound label is rebuilt deterministically afterwards. This keeps the model's label space small and pushes structure into code.
  • I2. The repair step realigns generated tokens to the reference tokens while respecting order and repeated tokens, defaults unmatched positions to "O", and maps unknown labels to "O". This turns noisy generations into usable, length-matched annotations.
  • I3. Results are reported twice, once under strict format compliance and once after lenient repair. This separates "the model knew the answer" from "the model wrote it correctly".
  • I4. Splitting the label set into separate narrow prompts and merging the answers did worse than one joint prompt. The authors attribute this to lost shared context and to errors that accumulate during merging.
  • I5. The model is asked to list the most probable tag first, so its output is a ranked candidate list rather than a single choice.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Prompts for our LLM teacher tiers could ask for simple, flat label pairs. Mova would then assemble richer composite labels itself, which makes the teacher's job easier and its errors easier to trace.
  • I2 (learn): When building silver data from LLM output, we could align its tokens to our own tokenizer and discard invented labels before witness parsers vote, so malformed rows never pollute training.
  • I3 (learn): Teacher-tier evaluations could keep a format-compliance score separate from a content score, showing whether a weak tier needs better prompting or better parsing of its replies.
  • I4 (frames): If an LLM teacher annotates roles, speech acts and coreference, asking for these jointly in one pass may beat separate per-layer queries that must be reconciled afterwards.
  • I5 (world): Entity typing for the world of the text could keep ranked type candidates, letting later reasoning fall back to a second option and explain why it switched.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Arab Center for Research and Policy StudiesDoha · Qatar8 works · 69 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Nancy Hamdan, Hadi Hamoud, Chadi Abou Chakra, Osama Rakan Al Mraikhat, Doha Albared and Fadi A. Zaraket. 2024. DRU at WojoodNER 2024: ICL LLM for Arabic NER. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.106

@inproceedings{hamdan2024dru,
  title = {{DRU at WojoodNER 2024: ICL LLM for Arabic NER}},
  author = {Nancy Hamdan, Hadi Hamoud, Chadi Abou Chakra, Osama Rakan {Al Mraikhat}, Doha Albared and Fadi A. Zaraket},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.106},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.106}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  2. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.
  3. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  4. Command R Team. 2024. Command R documentation. https://docs.cohere.com/docs/ command-r. Last accessed: 2024-05-15. ↗
  5. Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open arabic llm leaderboard. https://huggingface.co/spaces/ OALL/Open-Arabic-LLM-Leaderboard. ↗
  6. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic. Preprint, arXiv:2309.12053. ↗
  7. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa’ Omar. 2023. WojoodNER 2023: The first Arabic named entity recognition shared task. In Proceedings of Arabic-NLP 2023, pages 748–758, Singapore (Hybrid). Association for Computational Linguistics.
  8. Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, and Muhammad Elmadany, Abdel-Rahim Abdul-Mageed. 2024. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics.
  9. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. Preprint, arXiv:2205.09651. ↗
  10. Guochao Jiang, Zepeng Ding, Yuchen Shi, and Deqing Yang. 2024. P-icl: Point in-context learning for named entity recognition with large language models. Preprint, arXiv:2405.04960. ↗
  11. Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Arabic fine-grained entity recognition. Preprint, arXiv:2310.17333. ↗
  12. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? Preprint, arXiv:2202.12837. ↗
  13. Ying Mo, Jian Yang, Jiahao Liu, Shun Zhang, Jingang Wang, and Zhoujun Li. 2024. C-icl: Contrastive in-context learning for information extraction. Preprint, arXiv:2402.11254. ↗
  14. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2274–2285, Online. Association for Computational Linguistics. ↗
  15. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. Preprint, arXiv:cs/0306050. ↗
  16. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. Coarse lexical semantic annotation with supersenses: An Arabic case study. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 253–258, Jeju Island, Korea. Association for Computational Linguistics.
  17. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. Preprint, arXiv:2304.10428. ↗
  18. Meishan Zhang, Bin Wang, Hao Fei, and Min Zhang. 2024. In-context learning for few-shot nested named entity recognition. Preprint, arXiv:2402.01182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hamdan-2024-dru-wojoodner-2024-icl-llm/