Elamur Research⌕
☰
home / papers / 2021 / Dynamic Ensembles in Named Entity Recognition for Historical Arabic Texts

Dynamic Ensembles in Named Entity Recognition for Historical Arabic Texts

Paper · 2021 · Muhammad Majadly and Tomer Sagi

Summary

Most named entity recognition tools are built for modern English or trained on contemporary news and web Arabic. They transfer poorly to the classical Arabic of historical Islamic literature, where annotated data is scarce. The authors address this in two ways. First, they build a new annotated historical resource, the Bedaya Corpus. They use it to test whether training ML-based NER systems on period text helps more than training on modern text. Second, they borrow dynamic prediction from schema matching and pattern recognition. Voting, bagging and boosting fix the combination of base systems once. Here, instead, predictors estimate the quality of each base system's output without gold labels, and those estimates decide case by case which systems to trust. The authors compare the method with state-of-the-art Arabic NER and static ensembles on the new historical task. The dynamic ensemble performs best, reaching an F-score of about 0.8. This matters because it gives a practical approach for low-resource, out-of-period text, where no single tool is reliable on its own.

Key points

  • New resource: the Bedaya Corpus, a historical Arabic NER dataset built by the authors to make up for the lack of classical-Arabic training and test data.
  • The dynamic ensemble beats both state-of-the-art Arabic NER and static ensemble baselines on the historical task, with an F-score around 0.8.
  • A predictor is defined formally as a function that maps a NER result to a vector of real numbers without access to the expected labels. Predictors exist at several granularities, for example dataset-level and sentence-level.
  • The evaluation (Section 6.4) reports that training on Modern Standard Arabic data does not improve NER on historical Arabic text.
  • Context figures cited by the paper:
    • Best ANERcorp (modern Arabic) F1 is 89.9%, against 97.3% on a common English benchmark.
    • Speck and Ngomo's static ensembles cut English NER error by about 40% on average.

Interesting ideas

  • I1. Estimate how good each component's output is on this particular input, without gold labels, and let that estimate set its weight in the ensemble at run time instead of using one global weight per system. This is useful whenever several imperfect tools disagree on unfamiliar data.
  • I2. Compute quality signals at several levels of granularity, so a system can be trusted on one sentence and distrusted on the next. This captures local failures that dataset-wide averages hide.
  • I3. The paper's warning case: a usually reliable classifier that leans on capitalization will mislabel a sentence written entirely in capitals, and a static ensemble will still trust it heavily. Spotting inputs that break a component's core assumption is a cheap, general safeguard.
  • I4. Matching the training data to the period and register of the target text helped more than adding modern data. In this paper, more out-of-period data did not help.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data from several witness parsers, per-sentence quality predictors could decide whose analysis to trust. Examples are tree well-formedness, score margin, or agreement with the tagger.
  • I2 (tree): The perceptron parser and gparse could be combined per sentence, or even per arc, based on local confidence features, rather than one parser always winning.
  • I3 (tokens): The tagger could detect abnormal casing, such as headlines or all-caps text, and lower its trust in capitalization-driven PROPN decisions for those sentences.
  • I4 (learn): For tales and fables with archaic or literary style, Mova could build small in-register silver or gold sets and measure their effect against modern-news training. Unmatched data would serve as a negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Muhammad Majadly and Tomer Sagi. 2021. Dynamic Ensembles in Named Entity Recognition for Historical Arabic Texts. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{majadly2021dynamic,
  title = {{Dynamic Ensembles in Named Entity Recognition for Historical Arabic Texts}},
  author = {Muhammad Majadly and Tomer Sagi},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.12/}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosso. 2010. Using the yago ontology as a resource for the enrichment of Named Entities in Arabic WordNet. In Proceedings of The Seventh International Conference on Language Resources and Evaluation (LREC 2010) Workshop on Language Resources and Human Language Technology for Semitic Languages, pages 27–31. find
  2. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015. Polyglot-ner: Massive multilingual named entity recognition. In Proceedings of the society of industrial applied mathematics (SIAM ’15) International Conference on Data Mining, pages 586–594. SIAM. ↗
  3. Mohammad Al-Smadi, Saad Al-Zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer learning for arabic named entity recognition with deep neural networks. IEEE Access, 8:37736–37745. ↗
  4. Fahd Alotaibi and Mark Lee. 2014. A hybrid approach to features representation for fine-grained arabic named entity recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 984–995.
  5. Husamelddin AMN Balla and Sarah Jane Delany. 2020. Exploration of approaches to arabic named entity recognition. find
  6. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ruiz. 2007. Anersys: An Arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics (CICLing ’07), pages 143–153. Springer. find
  7. Ann Bies, Denise DiPersio, and Mohamed Maamouri. 2012. Linguistic resources for Arabic machine translation. John Benjamins, Amsterdam, The Netherlands. find
  8. Leo Breiman. 1996. Bagging predictors. Machine learning, 24(2):123–140.
  9. Leo Breiman. 2001. Random forests. Machine learning, 45(1):5–32. find
  10. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST ’11), 2(3):1–27. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, (MCS ’00), page 1–15, Berlin, Heidelberg. Springer-Verlag. find
  13. Maud Ehrmann, Matteo Romanello, Alex Flückiger, and Simon Clematide. 2020. Extended overview of clef hipe 2020: named entity processing on historical newspapers. In CLEF 2020 Working Notes. Conference and Labs of the Evaluation Forum, volume 2696. CEUR. find
  14. Asif Ekbal and Sivaji Bandyopadhyay. 2010. Named entity recognition using support vector machine: A language independent approach. International Journal of Electrical, Computer, and Systems Engineering, 4(2):155–170. ↗
  15. Yoav Freund and Robert E. Schapire. 1996. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning, ICML’96, page 148–156, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  16. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  17. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10– 18. ↗
  18. Trevor Hastie and Robert Tibshirani. 1998. Classification by pairwise coupling. In Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10, (NIPS ’97), page 507–513, Cambridge, MA, USA. MIT Press. ↗
  19. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. ↗
  20. Robert Hetzron. 1997. The Semitic Languages. Taylor & Francis. find
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  22. David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. 2013. Applied logistic regression, volume 398. John Wiley & Sons. find
  23. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. ↗
  24. Ridong Jiang, Rafael E Banchs, and Haizhou Li. 2016. Evaluating and combining name entity recognition systems. In Proceedings of the Sixth Named Entity Workshop, pages 21–27. Association for Computational Linguistics (ACL ’16). find
  25. George H. John and Pat Langley. 1995. Estimating continuous distributions in bayesian classifiers. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, (UAI ’95), page 338–345, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  26. Albert HR Ko, Robert Sabourin, and Alceu Souza Britto Jr. 2008. From dynamic classifier selection to dynamic ensemble selection. Pattern recognition, 41(5):1718–1731. ↗
  27. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  28. Niels Landwehr, Mark Hall, and Eibe Frank. 2005. Logistic model trees. Machine learning, 59(1-2):161– 205. ↗
  29. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by backpropagating errors. Nature, 323(6088):533–536. find
  30. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. ↗
  31. Tomer Sagi and Avigdor Gal. 2013. Schema matching prediction with applications to data source discovery and dynamic ensembling. The VLDB Journal, 22(5):689–710. ↗
  32. Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. Gcdt: A global context enhanced deep transition architecture for sequence labeling. arXiv preprint arXiv:1906.02437. ↗
  33. Mohamad Bagher Sajadi and Behrooz Minaei. 2017. Arabic named entity recognition using boosting method. In 2017 Artificial Intelligence and Signal Processing Conference (AISP ’17), pages 281–288, Shiraz, Iran. IEEE. find
  34. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60.
  35. Robert E Schapire. 1990. The strength of weak learnability. Machine learning, 5(2):197–227. ↗
  36. Khaled Shaalan. 2014. A survey of Arabic named entity recognition and classification. Computational Linguistics, 40(2):469–510. ↗
  37. Alireza Mansouri, Lilly Suriani Affendey, and Ali Mamat. 2008. Named entity recognition approaches. International Journal of Computer Science and Network Security IJCSNS, 8(2):339–344. find
  38. Khaled Shaalan and Hafsa Raza. 2009. Nera: Named entity recognition for Arabic. Journal of the American Society for Information Science and Technology, 60(8):1652–1663. ↗
  39. Slim Mesfar. 2007. Named entity recognition for Arabic using syntactic grammars. In International Conference on Application of Natural Language to Information Systems (NLDB ’07), pages 305–316. Springer. find
  40. René Speck and Axel-Cyrille Ngonga Ngomo. 2014. Ensemble learning for named entity recognition (iswc ’13). In International semantic web conference, pages 519–534. Springer. find
  41. Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics (EACL ’99).
  42. Djamel Mostefa, Mariama Laı̈b, Stéphane Chaudiron, Khalid Choukri, and G Chalendar. 2009. A multilingual named entity corpus for Arabic, English and French. MEDAR, 2009:2nd. find
  43. Nikunj Chandrakant Oza and Stuart Russell. 2001. Online ensemble learning. University of California, Berkeley. find
  44. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint arXiv:1808.08949. ↗
  45. J. Ross Quinlan. 1986. Induction of decision trees. Machine learning, 1(1):81–106. ↗
  46. J Ross Quinlan. 2014. C4. 5: programs for machine learning. Elsevier. find
  47. Xiang Ren, Jiaming Shen, Meng Qu, Xuan Wang, Zeqiu Wu, Qi Zhu, Meng Jiang, Fangbo Tao, Saurabh Sinha, David Liem, et al. 2017. Life-inet: A structured network-based knowledge exploration and analytics system for life sciences. In Proceedings of ACL 2017, System Demonstrations, pages 55–60. find
  48. Robert George Douglas Steel, James Hiram Torrie, et al. 1960. Principles and procedures of statistics. find
  49. Robert F Tate. 1954. Correlation between a discrete and a continuous variable. point-biserial correlation. The Annals of mathematical statistics, 25(3):603– 607. ↗
  50. Matthias Weidlich, Tomer Sagi, Henrik Leopold, Avigdor Gal, and Jan Mendling. 2013. Predicting the quality of process model matching. In Business Process Management (BPM ’13), pages 203–210. Springer. find
  51. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1792. ↗
  52. Norshuhani Zamin and Alan Oxley. 2011. Building a corpus-derived gazetteer for named entity recognition. In International Conference on Software Engineering and Computer Systems, pages 73–80. Springer. find
  53. GuoDong Zhou and Jian Su. 2002. Named entity recognition using an HMM-based chunk tagger. In proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 473–480. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/majadly-2021-dynamic-ensembles-named-entity-recognition/