Elamur Research
☰
home / papers / 2013 / Polyglot: Distributed Word Representations for Multilingual NLP

Polyglot: Distributed Word Representations for Multilingual NLP

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W13-3520.pdf

Summary

Most NLP systems rely on hand-engineered, language-specific features and are built mainly for English, so porting them to other languages is expensive. Learned word embeddings offer an alternative, but they are costly to train and had mostly been studied on English. The authors trained distributed word representations for 117 languages, each with more than 10,000 Wikipedia articles, using every language's Wikipedia as the corpus. The model follows the SENNA-style ranking approach. A neural network scores a window of words and receives a hinge-loss penalty unless it ranks the real window above a copy whose central word was randomly swapped, by a margin. Normalization is deliberately light: case is kept and digits become #. Vocabularies are capped at 100,000 frequent tokens. Optimizations contributed to Theano made the training feasible. A neural PoS tagger that uses only these embeddings as features, trained on the universal tagset, is competitive with the TnT tagger and even beats it for English and Danish. The public release gives researchers ready-made features for languages, including low-resource ones, where they lack expertise.

Key points

  • Resource: embeddings for 117 Wikipedia languages, with vocabularies of up to 100,000 words each. Out-of-vocabulary words are mapped to a single ⟨UNK⟩ token.
  • Because case is preserved, the English model keeps "Apple" near IT companies (e.g., Dell) and "apple" near fruits and vegetables.
  • PoS tagger setup:
    • window of 2 words on each side, hidden layer of 300, learning rate 0.3;
    • evaluated on the 12-tag universal tagset of Petrov et al. (2012);
    • German and Czech exceed 98.5% accuracy on known words;
    • Slovene, which had the least data, is weakest at 93.46%.
  • Replacing the embedding initialization with random vectors lowers accuracy for every language tested. The largest drops are Slovene (−2.68%, only 27,284 training examples) and Bulgarian (−2.01%); the smallest is English (−0.25%).
  • Main weaknesses are domain mismatch and handling of unknown words. Word-type coverage of the PoS datasets ranges from 60.68% (German) to 83.67% (Slovene), and the tagger uses no character-level information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

192 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 182 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alrfou2013polyglot,
  title = {{Polyglot: Distributed Word Representations for Multilingual NLP}},
  author = {Al-Rfou', Rami and Perozzi, Bryan and Skiena, Steven},
  booktitle = {Proceedings of the Seventeenth Conference on Computational Natural Language Learning},
  year = {2013},
  url = {https://aclanthology.org/W13-3520/}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Susana Afonso, Eckhard Bick, Renato Haber, and Diana Santos. 2002. Floresta sintá (c) tica”: a treebank for portuguese. In Proc. of the Third Intern. Conf. on Language Resources and Evaluation (LREC), pages 1698–1703. find
  2. Rami Al-Rfou’ and Steven Skiena. 2012. Speedread: A fast named entity recognition pipeline. In Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012), pages 53– 61, Mumbai, India, December. find
  3. Coling 2012 Organizing Committee. find
  4. Eduard Bejček, Jarmila Panevová, Jan Popelka, Pavel Straňák, Magda Ševčı́ková, Jan Štěpánek, and Zdeněk Žabokrtský. 2012. Prague Dependency Treebank 2.5 – a revisited version of PDT 2.0. In Proceedings of COLING 2012, pages 231–246, Mumbai, India, December. find
  5. The COLING 2012 Organizing Committee. find
  6. Yoshua Bengio and J-S Senecal. 2008. Adaptive importance sampling to accelerate training of a neural probabilistic language model. Neural Networks, IEEE Transactions on, 19(4):713–722. find
  7. Y. Bengio, H. Schwenk, J.S. Senécal, F. Morin, and J.L. Gauvain. 2006. Neural probabilistic language models. Innovations in Machine Learning, pages 137– 186. find
  8. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In International Conference on Machine Learning, ICML. find our card
  9. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral… find
  10. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Conference on Empirical Methods in Natural Language Processing, Sydney, Australia. find
  11. Léon Bottou. 1991. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nı̂mes 91, Nimes, France. EC2. find
  12. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The tiger treebank. In IN PROCEEDINGS OF THE WORK-SHOP ON TREEBANKS AND LINGUISTIC THEO-RIES, pages 24–41. find
  13. Thorsten Brants. 2000. Tnt: a statistical part-of-speech tagger. In Proceedings of the sixth conference on Applied natural language processing, pages 224–231. Association for Computational Linguistics. find our card
  14. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. find
  15. Minmin Chen, Zhixiang Xu, Kilian Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In John Langford and Joelle Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML-12), ICML ’12, pages 767–774. ACM, New York, NY, USA, Jul… find
  16. Yanqing Chen, Bryan Perozzi, Rami Al-Rfou’, and Steven Skiena. 2013. The expressive power of word embeddings. CoRR, abs/1301.3226. — [arXiv:1301.3226](https://arxiv.org/abs/1301.3226) find
  17. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML. find
  18. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493–2537, November. find our card
  19. Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In AISTATS. find
  20. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc Le, Mark Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Ng. 2012. Large scale distributed deep networks. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advance… find
  21. Sašo Džeroski, Tomaž Erjavec, Nina Ledinek, Petr Pajas, Zdenek Žabokrtsky, and Andreja Žele. 2006. Towards a slovene dependency treebank. In Proc. of the Fifth Intern. Conf. on Language Resources and Evaluation (LREC). find
  22. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11), volume 27, pages 97–110, June. find
  23. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependenc… find our card
  24. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459–1474, Mumbai, India, December. find
  25. The COLING 2012 Organizing Committee. find
  26. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In In Proc. ACL/HLT. find our card
  27. Matthias Trautner Kromann. 2003. The danish dependency treebank and the dtag treebank tool. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT), page 217. find
  28. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. find our card
  29. T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, and S. Khudanpur. 2010. Recurrent neural network based language model. Proceedings of Interspeech. find
  30. Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. Advances in neural information processing systems, 21:1081– 1088. find
  31. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252. find
  32. Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 216–225. Association for Computational Linguistics. find
  33. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. Talbanken05: A swedish treebank with phrase structure and dependency annotation. In Proceedings of the fifth International Conference on Language Resources and Evaluation (LREC), pages 1392–1395. find
  34. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  35. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. Association for Computational Linguistics. find
  36. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International C… find
  37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
  38. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 2002. Learning representations by backpropagating errors. Cognitive modeling, 1:213. find
  39. Lu Shuxiang. 2004. The Contemporary Chinese Dictionary (Xiandai Hanyu Cidian). Commercial Press. find
  40. Kiril Simov, Petya Osenova, Milena Slavcheva, Sia Kolkovska, Elisaveta Balabanova, Dimitar Doikoff, Krassimira Ivanova, Er Simov, and Milen Kouylekov. 2002. Building a linguistically interpreted corpus of bulgarian: the bultreebank. In In: Proceedings of LREC 2002, Canary Islands. find
  41. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24. find
  42. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  43. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 477–487. Asso… find our card
  44. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  45. Leonoor Van der Beek, Gosse Bouma, Rob Malouf, and Gertjan Van Noord. 2002. The alpino dependency treebank. Language and Computers, 45(1):8– 22. find