Elamur Research⌕
☰
home / papers / 2017 / Learning Language Representations for Typology Prediction

Learning Language Representations for Typology Prediction

Paper · 2017 · Chaitanya Malaviya, Graham Neubig and Patrick Littell

Summary

Typological databases such as WALS are sparsely filled, while NLP applications often need complete feature sets for the languages they handle. The authors test whether a neural model trained on parallel text can fill these gaps. They train a many-to-one NMT system from 1017 languages into English on Bible translations scraped from bible.com, then derive per-language vectors in three ways: a language-token embedding from a multilingual LSTM LM (LMVEC), the same kind of embedding from the NMT model (MTVEC), and a new representation that averages the encoder's LSTM cell states over all of a language's sentences (MTCELL). Per-feature logistic regression classifiers use these vectors to predict binary URIEL features covering syntax, phonology, and phonetic inventory. They are evaluated with and without a geographic/genetic 3-NN baseline. Combining both MT representations (MTBOTH) works best, beating both the majority-class rate and the neighbor baseline. The main gains are in syntactic features such as word order and case marking. This suggests that massively multilingual translation models encode recoverable typological knowledge that can help document under-described languages.

Key points

  • Data: a Bible corpus covering 1017 languages and about 20.6M training sentences. Vocabulary is shared through joint BPE with 32K merges. The model is a single-layer 512-dim LSTM built in DyNet, trained with Adam (lr 0.001) and dropout 0.5.
  • Targets: URIEL binary features, namely 103 syntax, 28 phonology, and 158 phonetic inventory features. Evaluation uses 10-fold cross-validation over languages.
  • Syntax accuracy without auxiliary features rises from 69.91 (majority class) to 77.11 with MTBOTH. With 3-NN neighbor features added, it goes from 83.07 to 86.33.
  • MTVEC beats LMVEC by roughly 2–3 points without auxiliary information, and MTCELL beats MTVEC. A paired bootstrap test finds the gains over the neighbor baseline significant for syntax and inventory but not for phonology.
  • The largest single gain is for numeral-after-noun word order (37.40 → 81.26). Some features, such as gender markers, case suffix, and negative affix, got slightly worse.

Interesting ideas

  • I1. Averaging an encoder's internal states to describe a whole input set. Instead of relying on a learned ID embedding, the method pools cell states across every token of every sentence. This kind of dynamic profile could characterize dialects, domains, or authors in other projects.
  • I2. Using cell states rather than hidden states. The authors found the gated hidden outputs carried less variance and information than the raw cell memory. This is a useful default when probing LSTMs.
  • I3. Translation alignment as supervision for representations. Translating into one shared target language gives a stronger signal than plain language modeling. With the geographic/genetic baseline added, the LM vectors' advantage disappeared while the MT vectors still helped. Parallel data in a pivot language such as English could similarly sharpen representations of Ukrainian and related low-resource varieties.
  • I4. Testing against a strong "neighbors" baseline. The comparison includes a geographic/genetic k-NN baseline and also combinations with it. This shows whether learned vectors add anything beyond simply knowing where a language is spoken, which is a good control for any geography-correlated prediction task.
  • I5. Checking where surprising gains come from. The unexpected improvements on phonology were traced to regionally clustered features, suggesting the model detects regions through syntax. Examining such side effects guards against over-interpreting what a probe "knows."

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Chaitanya Malaviya, Graham Neubig and Patrick Littell. 2017. Learning Language Representations for Typology Prediction. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1268

@inproceedings{malaviya2017learning,
  title = {{Learning Language Representations for Typology Prediction}},
  author = {Chaitanya Malaviya, Graham Neubig and Patrick Littell},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1268},
  url = {https://doi.org/10.18653/v1/d17-1268}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431–444.
  2. Gregory D.S. Anderson. 2013. The velar nasal. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  3. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
  4. Yevgeni Berzak, Roi Reichart, and Boris Katz. 2014. Reconstructing native language typology from foreign language usage. In Eighteenth Conference on Computational Natural Language Learning (CoNLL). find
  5. Ranko Bugarski. 1991. Contrastive analysis of terminology and the terminology of contrastive analysis. Languages in Contact und Contrast. Essays in Contact Linguistics/Edited by Vladimir Ivir and Damir Kalogjera.–Berlin, pages 73–82. find
  6. Reed Coke, Ben King, and Dragomir Radev. 2016. Classifying syntactic regularities for hundreds of languages. arXiv preprint arXiv:1603.08016. — [arXiv:1603.08016](https://arxiv.org/abs/1603.08016) find
  7. Chris Collins and Richard Kayne. 2011. Syntactic Structures of the World’s Languages. New York University, New York. find
  8. Bernard Comrie. 1989. Language Universals and Linguistic Typology: Syntax and Morphology. Blackwell, Oxford. find
  9. Joachim Daiber, Miloš Stanojević, and Khalil Sima’an. 2016. Universal reordering via linguistic typology. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3167–3176, Osaka, Japan. find
  10. The COLING 2016 Organizing Committee. find
  11. Hal Daumé III. 2009. Non-parametric Bayesian areal linguistics. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 593–601, Boulder, Colorado. Association for Computational Linguistics. find
  12. Hal Daumé III and Lyle Campbell. 2007. A Bayesian model for discovering typological implications. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 65–72, Prague, Czech Republic. Association for Computational Linguistics. find
  13. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  14. Noelia Ramón Garcı́a. 2002. Contrastive linguistics and translation studies interconnected: The corpus-based approach. Linguistica Antverpiensia, New Series–Themes in Translation Studies, (1). find
  15. Joseph Greenberg. 1963. Some universals of grammar with particular reference to the order of meaningful elements. In Joseph Greenberg, editor, Universals of Language, pages 110–113. MIT Press, London. find
  16. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. arXiv preprint arXiv:1611.04798.
  17. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2015. Glottolog 2.6. Max Planck Institute for the Science of Human History, Jena. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  19. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. arXiv preprint arXiv:1611.04558. find
  20. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
  21. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers… find
  22. M. Paul Lewis, Gary F. Simons, and Charles D. Fennig. 2015. Ethnologue: Languages of the World, Eighteenth edition. SIL International, Dallas, Texas. find
  23. William D Lewis and Fei Xia. 2008. Automatically identifying computationally relevant typological features. In Proceedings of the Third International Joint Conference on Natural Language Processing, Volume II, pages 685–690. find
  24. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational…
  25. Ian Maddieson. 2013a. Lateral consonants. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  26. Ian Maddieson. 2013b. Uvular consonants. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  27. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech, volume 2, page 3. find
  28. Steven Moran, Daniel McCloy, and Richard Wright. 2014. PHOIBLE Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  29. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 629–637. Association for Computational Linguistics.
  30. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980.
  31. Joanna Nichols. 1992. Linguistic Diversity in Space and Time. University of Chicago Press, Chicago. find
  32. Helen O’Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, and Anna Korhonen. 2016. Survey on the use of typological information in natural language processing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1297–1308, Osaka, Japa… find
  33. The COLING 2016 Organizing Committee. find
  34. Robert Östling. 2015. Word order typology through multilingual word alignment. In The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 205–211. find
  35. Robert Östling and Jörg Tiedemann. 2017. Continuous multilinguality with language vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 644–649, Valencia, Spain. Association for Computational Linguisti… find
  36. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444.
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
  38. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534, Austin, Texas. Association for Computational Linguistics.
  39. Oscar Täckström, Ryan McDonald, and Joakim Nivre. 2013. Target language adaptation of discriminative transfer parsers. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1061–1071, Atlanta, Georgi…
  40. Hiroya Takamura, Ryo Nagata, and Yoshifumi Kawasaki. 2016. Discriminative analysis of linguistic features for typological study. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find
  41. Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proceedings of the 2016 Conference of the North Am… find
  42. Yuan Zhang, Roi Reichart, Regina Barzilay, and Amir Globerson. 2012. Learning to map into a universal POS tagset. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1368–1378. Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malaviya-2017-learning-language-representations-typology-prediction/