Elamur Research⌕
☰
home / papers / 2017 / Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection

Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection

Paper · 2017 · Jian Ni, Georgiana Dinu and Radu Florian

Summary

The paper addresses building named entity recognition for a new language when no human-annotated NER data exist there, using a well-trained English system as the starting point. The authors propose two complementary weakly supervised routes. The first runs the English tagger on comparable (only possibly parallel) sentence pairs and carries the predicted tags over to the target side through word alignments. A heuristic, language-independent selection step then filters out noisy projected sentences before a target-language tagger is trained. The second route maps target-language word embeddings into the English embedding space, so the English neural tagger can label target text unchanged, without retraining. Two co-decoding schemes, one confidence-based and one rank-based, merge the outputs of both systems. On the CoNLL test sets for Spanish, Dutch and German, the combined systems outperform three earlier cross-lingual methods: those of Täckström and colleagues (2012), Nothman and colleagues (2013), and Tsai and colleagues (2016). They still trail fully supervised models by a wide margin. The work offers a strong, annotation-free baseline for extending NER to new languages.

Key points

  • Models used: a linear-chain CRF, a general-order MEMM, and two feedforward networks. NN1 takes word embeddings directly. NN2 adds a prototype "smoothing" layer and gives more balanced precision and recall.
  • CoNLL test F1 for the best co-decoding system (AP+NN1, confidence-based):
    • Spanish: 65.1, against 59.3, 61.0 and 60.6 for prior methods.
    • Dutch: 65.4, against at most 64.0 for prior methods.
    • German: 58.5, against at most 55.8 for prior methods.
    • Supervised baselines for comparison: 82.4, 84.5 and 71.8.
  • On the development data, exclude-O confidence-based co-decoding of AP and NN1 beat the better of the two single systems by 1.5–3.4 F1.
  • Weighting dictionary translation pairs by frequency when learning the embedding mapping raised NN1 by 2.2, 6.3 and 3.7 F1 on the Spanish, Dutch and German development sets, compared with uniform weights.
  • The data selection scheme helps most when alignment quality is poor and the projected labels are noisy.

Interesting ideas

  • I1. Filter projected labels instead of trusting them all. A cheap heuristic selection of good-quality projected sentences can make noisy comparable corpora usable. This is relevant for Ukrainian, where truly parallel data are limited.
  • I2. Map every language into one hub space. Projecting each language's embeddings into English with a seed dictionary means one source model serves many languages. A new language can be added without pair-specific training data.
  • I3. Weight translation pairs by frequency. When fitting a cross-lingual mapping, frequency weighting of dictionary entries can help noticeably. Keeping only the most frequent translation of each word gave similar gains, which is a simple trick for any bilingual-lexicon-based alignment.
  • I4. Combine two high-precision systems through co-decoding. Merging differently trained "views" of the same task by confidence or rank, while ignoring O predictions when resolving conflicts, can lift F1 above either system alone. This approach applies to other ensemble tagging setups.
  • I5. Avoid features that do not transfer. The neural taggers deliberately skip gazetteers and character-level features because these may be missing or behave differently in another language. This is a useful design principle for any cross-lingual transfer model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 worksNamed entity recognition · 101 works

Cite

Cite as Jian Ni, Georgiana Dinu and Radu Florian. 2017. Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1135

@inproceedings{ni2017weakly,
  title = {{Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection}},
  author = {Jian Ni, Georgiana Dinu and Radu Florian},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1135},
  url = {https://doi.org/10.18653/v1/p17-1135}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015. Polyglot-ner: Massive multilingual named entity recognition. In Proceedings of the 2015 SIAM International Conference on Data Mining. SIAM, Vancouver, British Columbia, Canada. https://doi.org/10.1137/1.9781611974010.66. — [doi:10.1137/1.9781611974010.66](https://doi.org/10.1137/1.9781611974010.66) ↗
  2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12:2493–2537. http://dl.acm.org/citation.cfm?id=1953048.2078186. ↗
  3. Mona Diab and Philip Resnik. 2002. An unsupervised method for word sense tagging using parallel corpora. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, ACL’02, pages 255–262. https://doi.org/10.3115/1073083.10731… ↗
  4. Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In Proceedings of Recent Advances in Natural Language Processing. Association for Computational Linguistics, pages 118–124. http://aclweb.org/anthology/R11-1017. ↗
  5. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Gothenburg, Sweden, pages 462– 4…
  6. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In Proceedings of the 32nd International Conference on Machine Learning. JMLR Workshop and Conference Proceedings, pages 748–756. http://jmlr.org/proceedings/papers/v37/… ↗
  7. Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Denver, Colorado, page… find
  8. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language…
  9. Abraham Ittycheriah and Salim Roukos. 2005. A maximum entropy word aligner for arabic-english machine translation. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. pages 89–96. http://aclweb.org/anthology/H05-1012. ↗
  10. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012.
  11. The COLING 2012 Organizing Committee, Mumbai, India, pages 1459–1474. http://www.aclweb.org/anthology/C12-1089. ↗
  12. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, ICML… find
  13. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Asso…
  14. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational…
  15. Andrew McCallum, Dayne Freitag, and Fernando C. N. Pereira. 2000. Maximum entropy markov models for information extraction and segmentation. In Proceedings of the Seventeenth International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, ICML’00, pages 591–598… find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR abs/1301.3781. http://arxiv.org/abs/1301.3781. ↗
  17. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. CoRR abs/1309.4168. http://arxiv.org/abs/1309.4168. ↗
  18. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes 30(1):3–26. Publisher: John Benjamins Publishing Company. https://doi.org/10.1075/li.30.1.03nad. ↗
  19. Jian Ni and Radu Florian. 2016. Improving multilingual named entity recognition with wikipedia entity type mapping. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1275–1284. https://doi.org/10.18653/v1/D16-1… ↗
  20. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses: An Introduction. John Wiley & Sons, Inc., New York, NY, USA. find
  21. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2013. Learning multilingual named entity recognition from wikipedia. Journal of Artificial Intelligence 194:151–175. https://doi.org/10.1016/j.artint.2012.03.006. ↗
  22. Sadegh Mohammad Rasooli and Michael Collins. 2015. Density-driven cross-lingual transfer of dependency parsers. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 328–338. https://doi.org/10.18653/v1/D15-1039. ↗
  23. E. Alexander Richman and Patrick Schone. 2008. Mining wiki resources for multilingual named entity recognition. In Proceedings of ACL-08: HLT. Association for Computational Linguistics, pages 1–9. http://aclweb.org/anthology/P08-1001. ↗
  24. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Com…
  25. Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. Association for Computational Linguistics, pages 219–228. https://doi.org/10.18653/v1/K16-1022. ↗
  26. Chen-Tse Tsai and Dan Roth. 2016. Cross-lingual wikification using multilingual embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 589– 598. ht…
  27. Ivan Vulić and Marie-Francine Moens. 2015. Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Langu… find
  28. Mengqiu Wang and D. Christopher Manning. 2014. Cross-lingual projected expectation regularization for weakly supervised learning. Transactions of the Association of Computational Linguistics 2:55–66. http://aclweb.org/anthology/Q14-1005. ↗
  29. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research. Association for Computational Linguistics, Stroudsburg, PA, U…
  30. Imed Zitouni and Radu Florian. 2008. Mention detection crossing the language barrier. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 600–609. http://aclweb.org/anthology/D08-1063. ↗
  31. Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of the Sixth Conference on Natural Language Learning -Volume 20. Association for Computational Linguistics, Stroudsburg, PA, USA, CONLL’02, pages 1–4. https://doi.o… ↗
  32. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 - Volume 4. Association for Computational Linguistics, Stroudsburg, PA,…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ni-2017-weakly-supervised-cross-lingual-named/