Elamur Research⌕
☰
home / papers / 2020 / A Semi-Supervised BERT Approach for Arabic Named Entity Recognition

A Semi-Supervised BERT Approach for Arabic Named Entity Recognition

Paper · 2020 · Chadi Helwe, Ghassan Dib, Mohsen Shamas et al.

Summary

The paper tackles Arabic named entity recognition (NER), which is hard for several reasons. Arabic script has no capital letters, many names double as ordinary nouns or adjectives, the morphology is rich, and annotated corpora are scarce. The authors restrict the task to person, location and organization names in BIO tagging. Their method is a teacher–student scheme modeled on Yalniz et al. (2019), built on the pretrained AraBERT model:

  1. A teacher is trained on the small gold corpus ANERCorp.
  2. The teacher fills in the missing tags of a large, automatically and only partially labeled dataset taken from earlier co-training work.
  3. Only sentences where the teacher is highly confident are kept.
  4. A student with the same architecture is trained on this filtered silver data.
  5. The student is then fine-tuned on the gold set.

On two Modern Standard Arabic benchmarks, AQMAR and NEWS, the approach beats MADAMIRA, FARASA, Deep Co-learning and a fully supervised AraBERT. On mostly Egyptian-dialect TWEETS it comes second. The work shows that cheap partial labels plus confidence filtering can substitute for scarce gold annotation.

Key points

  • Average F-measure on AQMAR was 65.5 for the semi-supervised AraBERT. This compares with 61.8 for Deep Co-learning, 61.5 for fully supervised AraBERT, 52.9 for FARASA and 29.2 for MADAMIRA.
  • On NEWS the semi-supervised model reached 78.6, against 74.1 for Deep Co-learning and 73.2 for the supervised baseline.
  • On TWEETS, Deep Co-learning scored 59.2 and the semi-supervised model 57.3, with supervised AraBERT at 54.0. The authors attribute the gap to AraBERT being pretrained on MSA text rather than noisy dialect.
  • Sentence selection works as follows:
    • Each unlabeled token gets the teacher's top-label probability over 7 BIO tags.
    • These are averaged per sentence.
    • A sentence is kept only if the average reaches τ = 0.95, a value tuned on validation data.
  • Training settings:
    • 20 epochs, batch size 32, dropout 0.2, Adam with early stopping.
    • Scoring used the CoNLL evaluation script.
    • Hardware was a single GTX 1080 Ti.
    • Organizations were consistently the weakest class.

Interesting ideas

  • I1. When rating how much to trust a teacher-annotated sentence, average confidence only over the tokens the teacher actually predicted, ignoring tags that were already given, so trusted labels do not inflate the score.
  • I2. Treat automatically obtained partial annotations as fixed anchors and let a trained model fill only the gaps, which turns a cheap, incomplete resource into a dense training set.
  • I3. Pretrain the student on filtered silver data first, then always finish with a fine-tuning pass on the small clean gold set; the authors stress that this final gold pass matters.
  • I4. Report results separately for each text register (formal news vs. dialectal, misspelled tweets), because a gain on clean text can hide a regression when pretraining and target domains differ.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers complete partially agreed silver trees, compute the acceptance score only over arcs that were disputed or newly filled, not over arcs already fixed by agreement.
  • I2 (world): entity mentions linked to Wikidata or WordNet could serve as fixed partial labels in raw text, with the tagger only labeling the remaining tokens to build silver entity data.
  • I3 (tree): gparse could be trained on large silver treebanks and then given a closing pass on gold UD treebanks, so that systematic silver errors are corrected last.
  • I4 (learn): evaluation suites could split scores by genre (fables, school tests, word problems, informal text) so that silver data drawn from one register is not mistaken for a general improvement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Chadi Helwe, Ghassan Dib, Mohsen Shamas and Shady Elbassuoni. 2020. A Semi-Supervised BERT Approach for Arabic Named Entity Recognition. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{helwe2020semi,
  title = {{A Semi-Supervised BERT Approach for Arabic Named Entity Recognition}},
  author = {Chadi Helwe, Ghassan Dib, Mohsen Shamas and Shady Elbassuoni},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  doi = {10.48448/6tcg-bv77},
  url = {https://doi.org/10.48448/6tcg-bv77}
}

References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sherief Abdallah, Khaled Shaalan, and Muhammad Shoaib. 2012. Integrating rule-based system with classification for arabic named entity recognition. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 311–322. Springer. find
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16. Association for Computational Linguistics, San Diego, California.
  3. Samir AbdelRahman, Mohamed Elarnaoty, Marwa Magdy, and Aly Fahmy. 2010. Integrated machine learning techniques for arabic named entity recognition. IJCSI, 7:27–36. find
  4. Ahmed Abdul-Hamid and Kareem Darwish. 2010. Simplified feature set for arabic named entity recognition. In Proceedings of the 2010 Named Entities Workshop, pages 110–115. Association for Computational Linguistics.
  5. Saleem Abuleil. 2004. Extracting names from arabic text for question-answering systems. In Coupling approaches, coupling media and coupling languages for information retrieval, pages 638–647. LE CENTRE DE HAUTES ETUDES INTERNATIONALES D’INFORMATIQUE DOCUMENTAIRE. find
  6. Riyad Al-Shalabi, Ghassan Kanaan, Bashar Al-Sarayreh, Khalid Khanfar, Ali Al-Ghonmein, Hamed Talhouni, and Salem Al-Azazmeh. 2009. Proper noun extracting algorithm for arabic language. In International conference on IT, Thailand. find
  7. Fahd Alotaibi and Mark G Lee. 2014. A hybrid approach to features representation for fine-grained arabic named entity recognition. In COLING, pages 984–995. 2007. Anercorp. http://www1.ccls.columbia.edu/ ybenajiba/downloads.html. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  9. Yassine Benajiba and Paolo Rosso. 2007. Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI, pages 1814–1823. find
  10. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. Springer. find
  11. Yassine Benajiba, Mona Diab, and Paolo Rosso. 2008a. Arabic named entity recognition using optimized feature sets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 284–293. Association for Computational Linguistics.
  12. Yassine Benajiba, Mona Diab, Paolo Rosso, et al. 2008b. Arabic named entity recognition: An svm-based approach. In Proceedings of 2008 Arab International Conference on Information Technology (ACIT), pages 16–18. ↗
  13. Yassine Benajiba, Imed Zitouni, Mona Diab, and Paolo Rosso. 2010. Arabic named entity recognition: using features extracted from noisy data. In Proceedings of the ACL 2010 conference short papers, pages 281–285. Association for Computational Linguistics.
  14. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example. In ACL (1), pages 1558–1567.
  15. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In LREC, volume 2, page 1.
  16. Ali Elsebai, Farid Meziane, and Fatma Zohra Belkredim. 2009. A rule based persons names arabic extraction system. Communications of the IBIMA, 11(6):53–59. find
  17. Mourad Gridach. 2016. Character-aware neural networks for arabic named entity recognition for social media. In Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016), pages 23–32.
  18. Chadi Helwe and Shady Elbassuoni. 2019. Arabic named entity recognition via deep co-learning. Artificial Intelligence Review, 52(1):197–215. ↗
  19. Rim Koulali and Abdelouafi Meziane. 2012. A contribution to arabic named entity recognition. In ICT and Knowledge Engineering (ICT & Knowledge Engineering), 2012 10th International Conference on, pages 46– 52. IEEE. find
  20. John Maloney and Michael Niv. 1998. Tagarab: a fast, accurate arabic name recognizer using high-precision morphological analysis. In Proceedings of the Workshop on Computational Approaches to Semitic Languages, pages 8–15. Association for Computational Linguistics.
  21. Slim Mesfar. 2007. Named entity recognition for arabic using syntactic grammars. In Natural Language Processing and Information Systems, pages 305–316. Springer. find
  22. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A Smith. 2012. Recall-oriented learning of named entities in arabic wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173. Association for Computational Linguistics.
  23. Mai Oudah and Khaled F Shaalan. 2012. A pipeline arabic named entity recognition using a hybrid approach. In COLING, pages 2159–2176.
  24. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101.
  25. Doaa Samy, Antonio Moreno, and Jose M Guirao. 2005. A proposal for an arabic named entity tagger leveraging a parallel corpus. In International Conference RANLP, Borovets, Bulgaria, pages 459–465. find
  26. Khaled Shaalan and Hafsa Raza. 2007. Person name entity recognition for arabic. In Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources, pages 17–24. Association for Computational Linguistics.
  27. Khaled Shaalan and Hafsa Raza. 2009. Nera: Named entity recognition for arabic. Journal of the American Society for Information Science and Technology, 60(8):1652–1663. ↗
  28. Khaled Shaalan. 2014. A survey of arabic named entity recognition and classification. Computational Linguistics, 40(2):469–510. ↗
  29. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. ↗
  30. I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/helwe-2020-semi-supervised-bert-approach-arabic/