Elamur Research⌕
☰
home / papers / 2012 / Recall-Oriented Learning of Named Entities in Arabic Wikipedia

Recall-Oriented Learning of Named Entities in Arabic Wikipedia

Paper · 2012 · Behrang Mohit, Nathan Schneider, Rishav Bhowmick et al.

Summary

The paper tackles named entity recognition in Arabic Wikipedia, where no labeled target-domain training data exists. Supervision is limited to news corpora (ACE, ANER). Arabic has rich morphology, no capitalization and mostly unwritten vowels. Wikipedia's entity mix also differs sharply from news, with many entities outside the usual person/organization/location (POL) classes. The authors built a small evaluation corpus (AQMAR) of 31 articles in four domains: history, science, sports and technology. Annotators followed ACE conventions for POL tags but could also invent a few article-specific categories, which were later folded into a generic MIS class. On the modeling side, they changed a structured perceptron's training loss so that missed entities cost more than spurious ones, making the learner deliberately "arrogant" about recall. They then applied self-training on unlabeled Wikipedia text. A news-trained baseline recalls poorly on Wikipedia. The recall-biased loss raises both recall and F1, and reusing the bias during self-training adds a small further gain. This matters because it shows a cheap, general recipe for cross-domain entity detection.

Key points

  • AQMAR corpus: about 3,000 sentences from 31 Arabic Wikipedia articles, tagged by two native speakers. It serves only for development and testing, while training uses ACE and ANER newswire.
  • Inter-annotator agreement was measured on the 4,750-word Gulf War article:
    • Cohen's κ was 0.86.
    • Entity boundary F1 was 94.0%, and category match F1 was 87.4%.
    • PER and MIS showed the weakest agreement, with PER disagreements mostly about titles and descriptors.
  • MIS entities are under 10% of mentions in news, but 53% in the development set and 37% in the test set.
  • The method is cost-augmented structured perceptron training with a recall-oriented cost controlled by one parameter, β. β was tuned on development data, and recall proved highly sensitive to its value.
  • Of 2,070 test entity types never seen in training, the systems recovered:
    • baseline: 450
    • reg/ROP: 588
    • ROP/none: 632
    • double-ROP: 717

Interesting ideas

  • I1. Penalizing false negatives more than false positives inside cost-augmented decoding biases a structured learner toward recall; the change amounts to one extra feature and costs little at inference.
  • I2. Annotators may coin their own article-specific entity classes, which are kept for analysis but merged into one MIS bucket for evaluation; this separates "is this an entity" from "what kind" when no fixed ontology fits.
  • I3. Self-training reused every automatically labeled sentence without confidence filtering, while keeping the same recall bias in the second round, and the gains held.
  • I4. Evaluation counted how many never-seen-in-training entity types each model recovered, showing that the recall gains were real generalization and not memorization.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The perceptron parser could use an asymmetric cost to stop it from under-predicting rare relations (e.g., appositions, flat names) when moving to out-of-domain text, with β tuned on a dev set.
  • I2 (world): The world builder could first register every entity mention, even without a type, and attach WordNet/Wikidata classes later, so unfamiliar technical or historical entities are not silently dropped.
  • I3 (learn): When retraining on silver data from witness parsers, it is worth testing an unfiltered run with a matching bias as a control against confidence-filtered runs.
  • I4 (learn): Mova's evaluations could report accuracy separately on unseen lemmas or entity types, as a guard against improvements that only reflect the training lexicon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer and Noah A. Smith. 2012. Recall-Oriented Learning of Named Entities in Arabic Wikipedia. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mohit2012recall,
  title = {{Recall-Oriented Learning of Named Entities in Arabic Wikipedia}},
  author = {Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer and Noah A. Smith},
  booktitle = {Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  url = {https://aclanthology.org/E12-1017/}
}

References

56 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdul-Hamid and Kareem Darwish. 2010. Simplified feature set for Arabic named entity recognition. In Proceedings of the 2010 Named Entities Workshop, pages 110–115, Uppsala, Sweden, July. Association for Computational Linguistics.
  2. Mohammed Attia, Antonio Toral, Lamia Tounsi, Monica Monachini, and Josef van Genabith. 2010. An automatically built named entity lexicon for Arabic. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias, editors, Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta, May. European Language Resources Association (ELRA).
  3. Bogdan Babych and Anthony Hartley. 2003. Improving machine translation quality with automatic named entity recognition. In Proceedings of the 7th International EAMT Workshop on MT and Other Language Technology Tools, EAMT ’03. find
  4. Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources, pages 10–18, Suntec, Singapore, August. Association for Computational Linguistics. find
  5. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́Ruiz. 2007. ANERsys: an Arabic named entity recognition system based on maximum entropy. In Alexander Gelbukh, editor, Proceedings of CICLing, pages 143–153, Mexico City, Mexio. Springer. find
  6. Yassine Benajiba, Mona Diab, and Paolo Rosso. 2008. Arabic named entity recognition using optimized feature sets. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 284–293, Honolulu, Hawaii, October. Association for Computational Linguistics.
  7. Philip K. Chan and Salvatore J. Stolfo. 1998. Toward scalable learning with non-uniform class and cost distributions: a case study in credit card fraud detection. In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, pages 164–168, New York City, New York, USA, August. AAAI Press. find
  8. Ciprian Chelba and Alex Acero. 2006. Adaptation of maximum entropy capitalizer: Little data can help a lot. Computer Speech and Language, 20(4):382– 399. ↗
  9. Massimiliano Ciaramita and Mark Johnson. 2003. Supersense tagging of unknown nouns in WordNet. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 168–175.
  10. Stephen Clark, James Curran, and Miles Osborne. 2003. Bootstrapping POS-taggers using unlabelled data. In Walter Daelemans and Miles Osborne, editors, Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 49–55. find
  11. Michael Collins. 2002. Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1– 8, Stroudsburg, PA, USA. Association for Computational Linguistics.
  12. Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708–716, Prague, Czech Republic, June. find
  13. James R. Curran, Tara Murphy, and Bernhard Scholz. 2007. Minimising semantic drift with Mutual Exclusion Bootstrapping. In Proceedings of PA-CLING, 2007. find
  14. Jeffrey Dalton, James Allan, and David A. Smith. 2011. Passage retrieval for incorporating global evidence in sequence labeling. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM ’11), pages 355–364, Glasgow, Scotland, UK, October. ACM. find
  15. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263, Prague, Czech Republic, June. Association for Computational Linguistics.
  16. Pedro Domingos. 1999. MetaCost: a general method for making classifiers cost-sensitive. Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 155–164. find
  17. Benjamin Farber, Dayne Freitag, Nizar Habash, and Owen Rambow. 2008. Improving NER in Arabic using a morphological tagger. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, and Daniel Tapias, editors, Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), pages 2509–2514, Marrakech, Morocco, May. European Language Resources Association (ELRA).
  18. Radu Florian, Hany Hassan, Abraham Ittycheriah, Hongyan Jing, Nanda Kambhatla, Xiaoqiang Luo, Nicolas Nicolov, and Salim Roukos. 2004. A statistical model for multilingual entity detection and tracking. In Susan Dumais, Daniel Marcu, and Salim Roukos, editors, Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, page 18, Boston, Massachusetts, USA, May. Association for Computational Linguistics. ↗
  19. Radu Florian, John Pitrelli, Salim Roukos, and Imed Zitouni. 2010. Improving mention detection robustness to noisy input. In Proceedings of EMNLP 2010, pages 335–345, Cambridge, MA, October. Association for Computational Linguistics. find
  20. Dayne Freitag. 2004. Trained named entity recognition using distributional clusters. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 262–269, Barcelona, Spain, July. Association for Computational Linguistics. find
  21. Kevin Gimpel and Noah A. Smith. 2010a. Softmax-margin CRFs: Training log-linear models with loss functions. In Proceedings of the Human Language Technologies Conference of the North American Chapter of the Association for Computational Linguistics, pages 733–736, Los Angeles, California, USA, June. find
  22. Kevin Gimpel and Noah A. Smith. 2010b. Softmax-margin training for structured log-linear models. Technical Report CMU-LTI-10-008, Carnegie Mellon University. http://www.lti.cs.cmu.edu/research/reports/2010/cmulti10008.pdf. ↗
  23. Cyril Grouin, Sophie Rosset, Pierre Zweigenbaum, Karn Fort, Olivier Galibert, and Ludovic Quintard. 2011. Proposal for an extension of traditional named entities: from guidelines to evaluation, an overview. In Proceedings of the 5th Linguistic Annotation Workshop, pages 92–100, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  24. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  25. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan and Claypool Publishers. ↗
  26. Ahmed Hassan, Haytham Fahmy, and Hany Hassan. 2007. Improving named entity translation by exploiting comparable and parallel corpora. In Proceedings of the Conference on Recent Advances in Natural Language Processing (RANLP ’07), Borovets, Bulgaria. find
  27. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL (HLT-NAACL), pages 57–60, New York City, USA, June. Association for Computational Linguistics. find
  28. Dirk Hovy, Chunliang Zhang, Eduard Hovy, and Anselmo Peas. 2011. Unsupervised discovery of domain-specific knowledge from text. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1466–1475, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  29. Jing Jiang and ChengXiang Zhai. 2006. Exploiting domain structure for named entity recognition. In Proceedings of the Human Language Technology Conference of the NAACL (HLT-NAACL), pages 74–81, New York City, USA, June. Association for Computational Linguistics. find
  30. Jun’ichi Kazama and Kentaro Torisawa. 2007. Exploiting Wikipedia as external knowledge for named entity recognition. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 698–707, Prague, Czech Republic, June. Association for Computational Linguistics.
  31. Chloe Kiddon and Yuriy Brun. 2011. That’s what she said: double entendre identification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 89–94, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  32. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain, July. Association for Computational Linguistics.
  33. LDC. 2005. ACE (Automatic Content Extraction) Arabic annotation guidelines for entities, version 5.3.3. Linguistic Data Consortium, Philadelphia. find
  34. Percy Liang, Hal Daumé III, and Dan Klein. 2008. Structure compilation: trading structure for features. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 592– 599, Helsinki, Finland. find
  35. Chris Manning. 2006. Doing named entity recognition? Don’t optimize for F 1 . http://nlpers.blogspot.com/2006/08/doing-named-entity-recognition-dont.html. ↗
  36. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159, New York City, USA, June. Association for Computational Linguistics.
  37. Rada Mihalcea. 2004. Co-training and self-training for word sense disambiguation. In HLT-NAACL 2004 Workshop: Eighth Conference on Computational Natural Language Learning (CoNLL-2004), Boston, Massachusetts, USA. find
  38. Einat Minkov, Richard Wang, Anthony Tomasic, and William Cohen. 2006. NER systems that suit user’s preferences: adjusting the recall-precision trade-off for entity extraction. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 93–96, New York City, USA, June. Association for Computational Linguistics. find
  39. Luke Nezda, Andrew Hickl, John Lehmann, and Sarmad Fayyaz. 2006. What in the world is a Shahab? Wide coverage named entity recognition for Arabic. In Proccedings of LREC, pages 41–46.
  40. Joel Nothman, Tara Murphy, and James R. Curran. 2009. Analysing Wikipedia and gold-standard corpora for NER training. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009), pages 612–620, Athens, Greece, March. Association for Computational Linguistics. find
  41. PediaPress. 2010. mwlib. http://code.pediapress.com/wiki/wiki/mwlib. ↗
  42. Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for accurate deterministic question parsing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 705–713, Cambridge, MA, October. Association for Computational Linguistics. find
  43. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155, Boulder, Colorado, June. Association for Computational Linguistics.
  44. Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. 2006. Subgradient methods for maximum margin structured learning. In ICML Workshop on Learning in Structured Output Spaces, Pittsburgh, Pennsylvania, USA. find
  45. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of ACL-08: HLT, pages 117–120, Columbus, Ohio, June. Association for Computational Linguistics.
  46. Satoshi Sekine, Kiyoshi Sudo, and Chikashi Nobata. 2002. Extended named entity hierarchy. In Proceedings of LREC. find
  47. Burr Settles. 2004. Biomedical named entity recognition using conditional random fields and rich feature sets. In Nigel Collier, Patrick Ruch, and Adeline Nazarenko, editors, COLING 2004 International Joint workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP) 2004, pages 107–110, Geneva, Switzerland, August. COLING.
  48. Khaled Shaalan and Hafsa Raza. 2008. Arabic named entity recognition from diverse text types. In Advances in Natural Language Processing, pages 440–451. Springer. find
  49. Mihai Surdeanu, David McClosky, Mason R. Smith, Andrey Gusev, and Christopher D. Manning. 2011. Customizing an information extraction system to a new domain. In Proceedings of the ACL 2011 Workshop on Relational Models of Semantics, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  50. Ben Taskar, Carlos Guestrin, and Daphne Koller. 2004. Max-margin Markov networks. In Sebastian Thrun, Lawrence Saul, and Bernhard Schölkopf, editors, Advances in Neural Information Processing Systems 16. MIT Press. find
  51. Antonio Toral, Elisa Noguera, Fernando Llopis, and Rafael Muñoz. 2005. Improving question answering using named entity recognition. Natural Language Processing and Information Systems, 3513/2005:181–191. find
  52. Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, and Yasemin Altun. 2005. Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research, 6:1453–1484, September. find
  53. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. ACE 2005 multilingual training corpus. LDC2006T06, Linguistic Data Consortium, Philadelphia. find
  54. Ralph Weischedel and Ada Brunstein. 2005. BBN pronoun coreference and entity type corpus. LDC2005T33, Linguistic Data Consortium, Philadelphia. ↗
  55. Dan Wu, Wee Sun Lee, Nan Ye, and Hai Leong Chieu. 2009. Domain adaptive bootstrapping for named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1523–1532, Singapore, August. Association for Computational Linguistics. find
  56. Tianfang Yao, Wei Ding, and Gregor Erbach. 2003. CHINERS: a Chinese named entity recognition system for the sports domain. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, pages 55–62, Sapporo, Japan, July. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohit-2012-recall-oriented-learning-named-entities/