Elamur Research⌕
☰
home / papers / 2024 / AREEj: Arabic Relation Extraction with Evidence

AREEj: Arabic Relation Extraction with Evidence

Paper · 2024 · Osama Rakan Al Mraikhat, Hadi Hamoud and Fadi A. Zaraket

Summary

The paper extends Arabic relation extraction so that each predicted link between two entities also comes with a short text span that justifies it. Each output is a source, a target, their entity types, a relation type and the supporting evidence phrase. To get training data, the authors enrich the Arabic part of SRED FM (2.9M relations) with evidence in three stages. First, ChatGPT-4 with a chain-of-thought prompt labels a stratified seed sample. Next, an mBART-based model, ArBartEv, is fine-tuned on those labels. Finally, ArBartEv labels the full corpus, producing ArSRED. They then train AREEj, an mBART seq2seq model that reads Arabic text and outputs relations together with evidence. On a human-revised Arabic RED FM test set, ArBartEv reaches .82 F1 for evidence. AREEj reaches .72 F1 for extraction, ahead of mREBEL at .69, although mREBEL was trained on far more multilingual data. This shows that evidence supervision can partly make up for scale and makes knowledge-graph edges explainable.

Key points

  • Seed sample (RDRS):
    • Only documents of 5–80 words were kept, which is about 89% of the corpus.
    • They were grouped into 2,526 bins by relation type plus source and target entity types, with 20 relations taken per bin.
    • This yielded 68,183 relations from 13,180 documents.
    • ChatGPT-4 gave evidence for 55,078 of them and "none" for 13,105.
  • ArBartEv and ArSRED:
    • ArBartEv maps (text, relation) to (relation, evidence).
    • It labelled 499,638 documents with 2.9M relations.
    • About 6% of its outputs were malformed, mostly from long documents, and were set to "none".
  • Test set: Arabic RED FM, human-revised, with 345 documents, 864 relations and 32 relation types.
  • Results:
    • AREEj extraction scores P .78, R .66, F1 .72, against mREBEL's P .89, R .56, F1 .69.
    • AREEj's evidence is correct 86% of the time.
    • Relation classification F1 is .78 for AREEj versus .71 for mREBEL.
    • AREEj finds 72 relations that mREBEL misses, while mREBEL finds 47 that AREEj misses.
    • AREEj detects 14 relation types that mREBEL does not, and mREBEL detects 4 that AREEj does not.
  • Training setup:
    • One RTX A6000 GPU.
    • Learning rate 5e-5, 5000 warm-up steps, batch size 16, 156,140 steps.
    • Arabic-only tokenizer, with entity and relation types as special tokens.

Interesting ideas

  • I1. Making the justifying text span a required part of every extracted relation turns each knowledge-graph edge into something a reader can check, and the paper suggests it also serves as a hint that raises recall.
  • I2. Spending an expensive LLM only on a stratified seed and then distilling it into a small seq2seq annotator for the full corpus keeps teacher cost low while still covering millions of items.
  • I3. The seed sample was built by binning on combinations of relation type and entity types and filling the smallest bins first, so rare configurations are guaranteed to appear.
  • I4. When the LLM returned too few answers for documents with many relations, stating in the prompt how many items were expected partly fixed the omissions.
  • I5. Evidence was evaluated with a cheap automatic filter first: no word overlap with the reference counts as a miss, and only overlapping cases go to a human. The paper also notes that the gold data missed some correct relations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Each frame role filled by prag could store the token span that licensed it, so an explanation of "who did what to whom" can point back to the exact words.
  • I2 (learn): LLM teacher tiers could label a small, carefully stratified seed, with a cheap local model trained to produce the bulk silver data.
  • I3 (learn): Silver-data selection could bin sentences by frame/VerbNet class plus argument types so rare constructions are not drowned out by frequent ones.
  • I4 (learn): Teacher prompts could state the expected number of frames or arguments to check, reducing silent drops in LLM-generated annotations.
  • I5 (learn): Evaluation could use automatic overlap as a first filter with targeted manual review, and track "correct but absent from gold" items so undercounted gold does not unfairly penalise the system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Arab Center for Research and Policy StudiesDoha · Qatar9 works · 69 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksInformation extraction · 175 works

Cite

Cite as Osama Rakan Al Mraikhat, Hadi Hamoud and Fadi A. Zaraket. 2024. AREEj: Arabic Relation Extraction with Evidence. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.6

@inproceedings{mraikhat2024areej,
  title = {{AREEj: Arabic Relation Extraction with Evidence}},
  author = {Osama Rakan {Al Mraikhat}, Hadi Hamoud and Fadi A. Zaraket},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.6},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.6}
}

References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  2. Youmi Ma, An Wang, and Naoaki Okazaki. 2023. DREEAM: Guiding attention with evidence for improving document-level relation extraction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1971–1983, Dubrovnik, Croatia. Association for Computational Linguistics. find
  3. Niama El Khbir, Nadi Tomeh, and Thierry Charnois. 2022. ArabIE: Joint entity, relation and event extraction for Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 331–345, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  5. Danish Pruthi, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Weakly- and semi-supervised evidence extraction. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3965–3970, Online. Association for Computational Linguistics. find
  6. Spence Green and Christopher D. Manning. 2010. Better Arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 394–402, Beijing, China. Coling 2010 Organizing Committee.
  7. Yasser Mohamed Saber, Hala Abdel-Galil, and Mohamed Abd El-Fatah Belal. 2022. Arabic ontology extraction model from unstructured text. Journal of King Saud University-Computer and Information Sciences, 34(8):6066–6076. ↗
  8. Kevin Huang, Peng Qi, Guangtao Wang, Tengyu Ma, and Jing Huang. 2021. Entity and evidence guided document-level relation extraction. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 307–315, Online. Association for Computational Linguistics. find
  9. Alessandro Seganti, Klaudia Firl ag, Helena Skowronska, Michał Satława, and Piotr Andruszkiewicz. 2021. Multilingual entity and relation extraction dataset and model. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1946–1955, Online. Association for Computational Linguistics. find
  10. Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370– 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  11. Nasrin Taghizadeh, Heshaam Faili, and Jalal Maleki. 2018. Cross-language learning for arabic relation extraction. Procedia Computer Science, 142:190– 197. find
  12. Pere-Lluís Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, and Roberto Navigli. 2023. Red fm : a filtered and multilingual relation extraction dataset. In Proc. of the 61st Annual Meeting of the Association for Computational Linguistics: ACL 2023, Toronto, Canada. Association for Computational Linguistics. find
  13. Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023. Revisiting relation extraction in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15566–15589, Toronto, Canada. Association for Computational Linguistics. find
  14. Ameen Jaber and Fadi A. Zaraket. 2017. Morphology-based entity and relational entity extraction framework for arabic. ArXiv, abs/1709.05700. ↗
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  16. Gehad Zakria, Mamdouh Farouk, Khaled Fathy, and Malak Makar. 2019. Relation extraction from arabic wikipedia. Indian Journal of Science and Technology, 12:01–06. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mraikhat-2024-areej-arabic-relation-extraction-evidence/