Elamur Research⌕
☰
home / papers / 2017 / Cross-lingual Character-Level Neural Morphological Tagging

Cross-lingual Character-Level Neural Morphological Tagging

Paper · 2017 · Ryan Cotterell and Georg Heigold

Summary

Most languages lack the thousands of annotated sentences that strong morphological taggers need. Cotterell and Heigold address this with transfer learning. They train a character-level recurrent tagger for a low-resource language together with one for a related high-resource language, such as Catalan paired with Spanish. The base model, from Heigold et al. (2017), builds word vectors with a character bidirectional LSTM, passes them through a sentence-level biLSTM, and predicts each word's full UD morphological tag independently. Each language is treated as a task in a multi-task setup, and all languages share one character embedding space. Three variants differ in how they let the model capture language-specific behaviour. Low-resource conditions were simulated on Universal Dependencies treebanks for 18 languages from four families, in single-source and multi-source settings. Joint training raised low-resource accuracy by as much as 30% and beat strong baselines. It needs no bitext or bilingual lexicon, and transfer worked better the closer the languages were.

Key points

  • The experiments cover 18 languages from four families using UD treebanks. The authors also release standard low-resource training splits of UD for future research.
  • Three transfer architectures extend the vanilla character-level biLSTM tagger. One uses a language-universal softmax, one adds joint language identification, and one uses language-specific components.
  • Gains on the low-resource language reach up to 30% accuracy. Accuracy on the high-resource language holds steady or slightly improves.
  • The models outperform MarMoT and the bitext-projection approach of Buys and Botha (2016). Nine languages enter the F1 comparison: Swedish (sv), Portuguese (pt), Polish (pl), Italian (it), Finnish (fi), Spanish (es), Danish (da), Czech (cs) and Bulgarian (bg).
  • The method still needs a small amount of annotated target-language data. In return, it avoids the bilingual lexicons and aligned bitext that projection methods depend on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States247 works · 106,235 citations🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany79 works · 29,970 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksMorphology · 420 works

Cite

Cite as Ryan Cotterell and Georg Heigold. 2017. Cross-lingual Character-Level Neural Morphological Tagging. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1078

@inproceedings{cotterell2017cross,
  title = {{Cross-lingual Character-Level Neural Morphological Tagging}},
  author = {Ryan Cotterell and Georg Heigold},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1078},
  url = {https://doi.org/10.18653/v1/d17-1078}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR).
  2. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 349–359, Lisbon, Portugal. Association for Computati…
  3. Jan Buys and Jan A. Botha. 2016. Cross-lingual morphological tagging for low-resource languages. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1954–1964, Berlin, Germany. Association for Computational Linguistics. find
  4. Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41–75. find
  5. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011a. Torch7: A Matlab-like environment for machine learning. In BigLearn, NIPS Workshop. find
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011b. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
  7. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  8. The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, Canada. Association for Computational Linguistics.
  9. Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL), pages 334– 343, Beijing, China. Association for Computational Linguistics. find
  10. Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (N… find
  11. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages…
  12. Victoria Fossum and Steven P. Abney. 2005. Automatically inducing a part-of-speech tagger by projecting from multiple source languages across aligned corpora. In Second International Joint Conference on Natural Language Processing (IJCNLP), pages 862– 873. find
  13. Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 138–147, Atlanta, Georgia. Associati…
  14. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6):602–610. find
  15. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  16. The SIGMORPHON 2016 shared task— morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics. find
  17. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language…
  18. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL), pages 600–609, Portland, Oregon, USA. Association for Computa…
  19. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI), pages 2734–2740. find
  20. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Jan Hajič and Barbora Hladká. 1998. Tagging inflective languages: Prediction of morphological categories for…
  21. Georg Heigold, Guenter Neumann, and Josef van Genabith. 2017. An extensive empirical evaluation of character-based morphological tagging for 14 languages. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 505–513, Valencia, S… find
  22. Georg Heigold, Günter Neumann, and Josef van Genabith. 2016. Neural morphological tagging from characters for morphologically rich languages. CoRR, abs/1606.06640. — [arXiv:1606.06640](https://arxiv.org/abs/1606.06640) find
  23. Georg Heigold, Vincent Vanhoucke, Andrew Senior, Patrick Nguyen, Marc’Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean. 2013. Multilingual acoustic models using distributed deep neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8619–8623. I… find
  24. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  25. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. CoRR, abs/1611.0455…
  26. Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017. One-shot neural cross-lingual transfer for paradigm completion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. Association for Computational Linguistics. find
  27. Aleksandr E. Kibrik. 1998. Archi (Caucasian – Daghestanian). In The Handbook of Morphology, pages 455–476. Blackwell Oxford. find
  28. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML), pages 282–289. find
  29. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proc…
  30. Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589. find
  31. Thomas Müller, Ryan Cotterell, Alexander Fraser, and Hinrich Schütze. 2015. Joint lemmatization and morphological tagging with Lemming. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2268–2274, Lisbon, Portugal. Association for Computationa…
  32. Thomas Müller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 322–332, Seattle, Washington, USA. Association for Computational Linguistics. find
  33. Frederick J. Newmeyer. 2007. Linguistic typology requires crosslinguistic formal categories. Linguistic Typology, 11(1):133–157. find
  34. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on…
  35. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  36. Kemal Oflazer and Ìlker Kuruöz. 1994. Tagging and morphological disambiguation of Turkish text. In Proceedings of the Fourth Conference on Applied Natural Language Processing, pages 144–149. Association for Computational Linguistics. find
  37. Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), pages 2089–2096.
  38. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 412–418, Berlin, Germany. Associati…
  39. Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 623–633, San Diego, C… find
  40. Cı́cero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31th International Conference on Machine Learning (ICML), pages 1818–1826. find
  41. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958.
  42. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In 13th Annual Conference of the International Speech Communication Association (INTER-SPEECH), pages 194–197. find
  43. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 3104–3112.
  44. John Sylak-Glassman, Christo Kirov, David Yarowsky, and Roger Que. 2015. A language-independent feature schema for inflectional morphology. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Proces…
  45. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 477–…
  46. Tijmen Tieleman and Geoff Hinton. 2012. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. find
  47. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. WSABIE: scaling up to large vocabulary image annotation. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI), pages 2764–2770. find
  48. David Yarowsky and Grace Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Language Technologies 2001: The Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  49. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Václava Kettnerová, Zdenka Uresova,… find
  50. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics.
  51. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (NIPS), pages 649–657.
  52. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 30–34, San Diego, California. Association for Computational Linguistic…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cotterell-2017-cross-lingual-character-level-neural/