Elamur Research⌕
☰
home / papers / 2020 / Embed More Ignore Less (EMIL): Exploiting Enriched Representations for Arabic NLP

Embed More Ignore Less (EMIL): Exploiting Enriched Representations for Arabic NLP

Paper · 2020 · Ahmed Younes and Julie Weeds

Summary

The paper tackles Arabic sequence labelling (NER and POS tagging). Arabic NLP preprocessing often normalizes text, and most modern text is written without diacritics, the short-vowel marks that carry morphological, syntactic and semantic cues. The authors argue against discarding such signals and propose EMIL ("Embed More, Ignore Less"). They restore diacritics automatically with the Shakkala diacritizer, then give a Bi-LSTM-CRF tagger a separate diacritic embedding layer in addition to its word and character embeddings. For NER they go one step further. Their own diacritic-aware POS tagger labels the NER corpora, and those predicted tags become a fourth embedding channel. They test the approach on seven datasets, four for NER and three for POS. Adding diacritic information helps on every dataset, and predicted POS tags bring further gains for NER, even though both signals are produced automatically and contain errors. The study shows that cheap, automatically inferred linguistic signals can be worth more than the noise they carry.

Key points

  • The baseline is the character-aware Bi-LSTM-CRF of Gridach (2016). EMIL extends it with diacritic embeddings and, for NER, a four-layer combination of word, character, diacritic and POS embeddings.
  • Diacritics come from Shakkala, a Bi-LSTM diacritizer with character embeddings, applied to undiacritized input. POS tags for the NER data come from the authors' diacritic-enhanced tagger.
  • In the ablation study the full EMIL model reaches F1 0.929. Freezing the embeddings instead of training them drops F1 to 0.591, a significant difference (paired test, 5% level). The chosen setup combines:
    • pre-trained word embeddings,
    • randomly initialized POS embeddings,
    • separately trained character and diacritic embeddings.
  • On ANERCorp, total NER errors fall from 5282 (baseline) to 3139 (with diacritics) and then to 1074 (four-layer model).
  • The largest error reductions are in the two tag-ambiguity categories: "dominant tag" errors fall from 701 to 117 and "counter-dominant tag" errors from 635 to 60.

Interesting ideas

  • I1. Instead of normalizing away information the surface text does not show, infer it with an existing tool and feed it to the model as its own embedding channel next to words and characters.
  • I2. Stacking imperfect predicted layers can still pay off, and one layer can partly repair another's mistakes; in the paper, predicted POS sometimes corrected wrong diacritics.
  • I3. The error analysis sorts mistakes into boundary errors, rare words, dominant-tag errors and counter-dominant errors, and the last group often turned out to be gold-standard mistakes rather than model mistakes.
  • I4. Each architectural choice is backed by an ablation with a paired significance test, including a frozen-versus-trainable embedding comparison.
  • I5. Diacritics on the last letter encode syntactic function, so restoring them gives the model indirect syntax at no annotation cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Signals such as original casing, contraction splits or punctuation context could be kept as separate feature channels for the tagger rather than erased during normalization.
  • I2 (tree): Predicted morphology and lemma features could feed the perceptron parser and gparse as embeddings, with ablations checking whether each layer adds value despite its noise.
  • I3 (learn): Applying the same buckets to disagreements between witness parsers and gold treebanks would separate model weaknesses from likely annotation errors worth repairing.
  • I4 (learn): Every new feature channel should be judged against its removal under a paired significance test, not by a single headline score.
  • I5 (tokens): Hidden-but-recoverable grammatical cues in English, such as an inferred case role for pronouns or a predicted verb form, could similarly be restored first and passed on as explicit evidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksPart-of-speech tagging · 126 works

Cite

Cite as Ahmed Younes and Julie Weeds. 2020. Embed More Ignore Less (EMIL): Exploiting Enriched Representations for Arabic NLP. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{younes2020embed,
  title = {{Embed More Ignore Less (EMIL): Exploiting Enriched Representations for Arabic NLP}},
  author = {Ahmed Younes and Julie Weeds},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.13/}
}

References

53 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Ahmed Abdelali, Ali Elkahky, Hamdy Mubarak, Kareem Darwish, Mohammed Attia, and Younes Samih. 2019. Pos tagging for improving code-switching identification in arabic. find
  3. Samir AbdelRahman, Mohamed Elarnaoty, Marwa Magdy, and Aly Fahmy. 2010. Integrated machine learning techniques for arabic named entity recognition. IJCSI, 7:27–36. find
  4. Ahmed Abdul-Hamid and Kareem Darwish. 2010. Simplified feature set for arabic named entity recognition. In Proceedings of the 2010 Named Entities Workshop, pages 110–115. Association for Computational Linguistics.
  5. Rabab Ali Abumalloh, Hassan Maudi Al-Sarhan, Othman Ibrahim, and Waheeb Abu-Ulbeh. 2016. Arabic part-of-speech tagging. Journal of Soft Computing and Decision Support Systems, 3(2):45–52. find
  6. Ahmad Al-Sallab, Ramy Baly, Hazem Hajj, Khaled Bashir Shaban, Wassim El-Hajj, and Gilbert Badaro. 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):1–20. find
  7. Fatma Al Shamsi and Ahmed Guessoum. 2006. A hidden markov model-based pos tagger for arabic. In Proceeding of the 8th International Conference on the Statistical Analysis of Textual Data, France, pages 31–42. find
  8. Mohammed Nadher Abdo Ali, Guanzheng Tan, and Aamir Hussain. 2019. Boosting arabic named-entity recognition with multi-attention layer. IEEE Access, 7:46575–46582. ↗
  9. Fahd Alotaibi and Mark Lee. 2014. A hybrid approach to features representation for fine-grained arabic named entity recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 984–995.
  10. Sawsan Alqahtani, Hanan Aldarmaki, and Mona Diab. 2019. Homograph disambiguation through selective diacritic restoration. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 49–59, Florence, Italy, August. Association for Computational Linguistics. find
  11. Shihadeh Alqrainy. 2008. A morphological-syntactical analysis approach for arabic textual tagging. find
  12. Khwlah Alrajhi and Mohammed A ELAffendi. 2019. Automatic arabic part-of-speech tagging: Deep learning neural lstm versus word2vec. International Journal of Computing and Digital Systems, 8(03):307–315. find
  13. Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2017. Arabic machine transliteration using an attention-based encoder-decoder model. Procedia Computer Science, 117:287–297. find
  14. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  15. Ramy Baly, Hazem Hajj, Nizar Habash, Khaled Bashir Shaban, and Wassim El-Hajj. 2017. A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in arabic. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):1–21. ↗
  16. Zerrouki Barqawi. 2017. Shakkala, arabic text vocalization. find
  17. Yassine Benajiba and Paolo Rosso. 2007. Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI. find
  18. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. Springer. find
  19. Yassine Benajiba, Mona Diab, Paolo Rosso, et al. 2008a. Arabic named entity recognition: An svm-based approach. In Proceedings of 2008 Arab International Conference on Information Technology (ACIT), pages 16–18. Association of Arab Universities Amman, Jordan. ↗
  20. Yassine Benajiba, Mona Diab, Paolo Rosso, et al. 2008b. Arabic named entity recognition: An svm-based approach. In Proceedings of 2008 Arab International Conference on Information Technology (ACIT), pages 16–18. Association of Arab Universities Amman, Jordan. ↗
  21. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, and Mohamed Eldesouki. 2017. Arabic pos tagging: Don’t abandon feature engineering just yet. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 130–137.
  22. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567.
  23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  24. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritization using deep neural networks. arXiv preprint arXiv:1905.01965. ↗
  25. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
  26. Rana Forsati and Mehrnoush Shamsfard. 2014. Hybrid pos-tagging: A cooperation of evolutionary and statistical approaches. Applied Mathematical Modelling, 38(13):3193–3211. ↗
  27. Mourad Gridach. 2016. Character-aware neural networks for arabic named entity recognition for social media. In Proceedings of the 6th workshop on South and Southeast Asian natural language processing (WSSANLP2016), pages 23–32.
  28. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological analysis and disambiguation for dialectal arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 426–432.
  29. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  30. Jan Hajic, Otakar Smrz, Petr Zemánek, Jan Šnaidauf, and Emanuel Beška. 2004. Prague arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools, pages 110–117. find
  31. Ayoub Kadim and Azzeddine Lazrek. 2018. Parallel hmm-based approach for arabic part of speech tagging. Int. Arab J. Inf. Technol., 15(2):341–351. find
  32. Muhammad Khalifa and Khaled Shaalan. 2019. Character convolutions for arabic named entity recognition with long short-term memory networks. Computer Speech & Language, 58:335–346. ↗
  33. Shereen Khoja. 2001. Apt: Arabic part-of-speech tagger. In Proceedings of the Student Workshop at NAACL, pages 20–25. find
  34. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. ↗
  35. Slim Mesfar. 2007. Named entity recognition for arabic using syntactic grammars. In International Conference on Application of Natural Language to Information Systems, pages 305–316. Springer. find
  36. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  37. Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Younes Samih, and Hassan Sajjad. 2019a. A system for diacritizing four varieties of arabic. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 217–222.
  38. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019b. Highly effective arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395.
  39. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. ↗
  40. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland, May. European Language Resources Association (ELRA).
  41. Khaled Shaalan and Hafsa Raza. 2007. Person name entity recognition for arabic. In Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources, pages 17–24. Association for Computational Linguistics.
  42. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Abdel Monem. 2019. Challenges in arabic natural language processing. Computational Linguistics. ↗
  43. Khaled Shaalan. 2014. A survey of arabic named entity recognition and classification. Computational Linguistics, 40(2):469–510. ↗
  44. Jiabao Sheng, Aishan Wumaier, and Zhe Li. 2020. Poise: Efficient cross-domain chinese named entity recognization via transfer learning. Symmetry, 12(10):1673. ↗
  45. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  46. Srishti Vashishtha and Seba Susan. 2019. Fuzzy rule based unsupervised sentiment analysis from social media posts. Expert Systems with Applications, 138:112834. ↗
  47. Peilu Wang, Yao Qian, Frank K Soong, Lei He, and Hai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. arXiv preprint arXiv:1510.06168. ↗
  48. Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. ↗
  49. Jabar H Yousif and T Sembok. 2005. Arabic part-of-speech tagger based neural networks. In proceedings of International Arab Conference on Information Technology ACIT2005, ISSN, volume 857. find
  50. Jabar H Yousif and T Sembok. 2006. Design and implement an automatic neural tagger based arabic language for nlp applications. Asian Journal of Information Technology, 5(7):784–789. find
  51. Jabar Hassan Yousif and Tengku Mohd Tengku Sembok. 2008. Arabic part-of-speech tagger based support vectors machines. In 2008 International Symposium on Information Technology, volume 3, pages 1–7. IEEE. find
  52. Imad Zeroual and Lakhouaja Abdelhak. 2016. Adapting a decision tree based tagger for arabic. In 2016 International Conference on Information Technology for Organizations Development (IT4OD), pages 1–6. IEEE. find
  53. Inès Zribi, Inès Kammoun, Mariem Ellouze, L Belguith, and Philippe Blache. 2016. Sentence boundary detection for transcribed tunisian arabic. Bochumer Linguistische Arbeitsberichte, 323. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/younes-2020-embed-more-ignore-less-emil/