Elamur Research
☰
home / papers / 2012 / Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure

Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N12-1052.pdf

Summary

Word cluster features learned from unlabeled text are known to help structured prediction. However, the evidence came mostly from English, and no one had combined such features with cross-lingual transfer. The authors address both gaps.

First, they induce monolingual clusters with the predictive class bigram model of Uszkoreit and Brants (2008) on large newswire corpora. They add cluster-based templates to a transition-based dependency parser and a CRF named-entity tagger, testing 13 languages for parsing and 4 for NER.

Second, they propose two ways to obtain clusters that are consistent across languages:

  • projecting English clusters to a target language through word alignments;
  • jointly clustering monolingual data in both languages, with alignments from parallel data acting as constraints.

These cross-lingual clusters are added to delexicalized direct-transfer models trained on English, following McDonald et al. (2011). In the supervised setting, cluster features never lowered accuracy. In transfer, they cut relative parsing error by up to 13% and NER error by up to 26%. The work shows that semi-supervised features and cross-lingual transfer can be combined, offering a cheap route to tools for resource-poor languages.

Key points

  • Clustering setup: 256 clusters over the 1 million most frequent word types per language, learned with the distributed exchange algorithm. DA had the smallest amount of unlabeled newswire, at 0.8 billion tokens, while EN had the largest, at 121.6 billion tokens.
  • Supervised parsing: average LAS rose from 83.0 to 84.0 across 13 languages, with gains significant everywhere except FR and NL.
    • Parser: arc-eager, beam of 8, averaged structured perceptron, features of Zhang and Nivre (2011).
    • Treebanks: CoNLL 2006/2007, Penn WSJ Treebank, French Treebank, Sejong Korean Treebank, SynTagRus, Penn Chinese Treebank v6.
  • Monolingual NER: on the CoNLL 2002/2003 development sets for DE, EN, ES and NL, average F1 improved from 76.3 to 83.0 with clusters.
  • Direct-transfer NER from English (test set): average F1 was 39.1 without clusters, 50.2 with projected clusters, and 52.7 with X-LINGUAL CLUSTERS. This is a 22% average error reduction, reaching 26% for ES. For entity-boundary detection alone, F1 rose from 68.1 to 75.8.
  • Word alignments: produced with the dual decomposition aligner of DeNero and Macherey (2011) on 10.5–12.1 million sentences of aligned web data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition · Dependency parsing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tackstrom2012cross,
  title = {{Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure}},
  author = {Täckström, Oscar and McDonald, Ryan and Uszkoreit, Jakob},
  booktitle = {Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2012},
  url = {https://aclanthology.org/N12-1052/}
}

References

54 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne Abeillé and Nicolas Barrier. 2004. Enriching a french treebank. In Proceedings of LREC. find
  2. Juri Apresjan, Igor Boguslavsky, Boris Iomdin, Leonid Iomdin, Andrei Sannikov, and Victor Sizov. 2006. A syntactically and semantically tagged corpus of russian: State of the art and prospects. In Proceedings of LREC. find
  3. Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology, 6(3):1–26. find
  4. Taylor Berg-Kirkpatrick and Dan Klein. 2010. Phylogenetic grammar induction. In Proceedings of ACL. find
  5. Daniel M. Bikel, Richard Schwartz, and Ralph M. Weischedel. 1999. An algorithm that learns what’s in a name. Machine Learning, 34(1):211–231. find
  6. Igor Boguslavsky, Svetlana Grigorieva, Nikolai Grigoriev, Leonid Kreidlin, and Nadezhda Frid. 2000. Dependency treebank for Russian: Concept, tools, types of information. In Proceedings of COLING. find
  7. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18:467–479. find
  8. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL. find our card
  9. Marie Candito, Benoı̂t Crabbé, and Pascal Denis. 2010. Statistical french dependency parsing: treebank conversion and first results. In Proceedings of LREC. find
  10. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL. find our card
  11. Alexander Clark. 2003. Combining distributional and morphological information for part of speech induction. In Proceedings of EACL. find
  12. Shay B. Cohen and Noah A. Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of NAACL. find
  13. Shay B. Cohen, Dipanjan Das, and Noah A. Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proceedings of EMNLP. find
  14. Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In Proceedings of ACL. find
  15. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of ACL. find
  16. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML. find
  17. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of ACL-HLT. find
  18. Marie-Catherine De Marneffe, Bill MacCartney, and Chris D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC. find our card
  19. John DeNero and Klaus Macherey. 2011. Model-based aligner combination using dual decomposition. In Proceedings of ACL-HLT. find
  20. Paramveer Dhillon, Dean Foster, and Lyle Dean. 2011. Multi-view learning of word embeddings via cca. In Proceedings of NIPS. find
  21. Manaal Faruqui and Sebastian Padó. 2010. Training and evaluating a german named entity recognizer with semantic generalization. In Proceedings of KONVENS. find
  22. Dayne Freitag. 2004. Trained named entity recognition using distributional clusters. In Proceedings of EMNLP. find
  23. Jennifer Gillenwater, Kuzman Ganchev, João Graça, Fernando Pereira, and Ben Taskar. 2010. Sparsity in dependency grammar induction. In Proceedings of ACL. find
  24. Gholamreza Haffari, Marzieh Razavi, and Anoop Sarkar. 2011. An ensemble model that combines syntactic and semantic clustering for discriminative dependency parsing. In Proceedings of ACL. find
  25. Chung-hye Han, Na-Rare Han, Eon-Suk Ko, and Martha Palmer. 2002. Development and evaluation of a korean treebank and its application to nlp. In Proceedings of LREC. find
  26. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(03):311–325. find
  27. Dan Klein and Chris D. Manning. 2004. Corpus-based induction of syntactic structure: models of dependency and constituency. In Proceedings of ACL. find
  28. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proceedings of ACL-HLT. find our card
  29. Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. Morgan & Claypool Publishers. find
  30. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of ACL-IJCNLP, pages 1030–1038. find
  31. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19(2):313–330. find our card
  32. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of EMNLP. find our card
  33. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In Proceedings of HLT-NAACL. find
  34. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML. find
  35. Tahira Naseem, Harr Chen, Regina Barzilay, and Mark Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of EMNLP. find
  36. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  37. The CoNLL 2007 shared task on dependency parsing. In Proceedings of EMNLP-CoNLL. find
  38. Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553. find our card
  39. Franz Josef Och. 1999. An efficient method for determining bilingual word classes. In Proceedings of EACL. find
  40. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Now Publishers Inc. find
  41. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. In ArXiv:1104.2086. find
  42. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of EMNLP. find
  43. Noah A. Smith. 2011. Linguistic Structure Prediction. Morgan & Claypool Publishers. find
  44. Anders Søgaard. 2011. Data point selection for cross-language adaptation of dependency parsers. In Proceedings of ACL. find
  45. Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. 2011. Lateen EM: Unsupervised training with multiple objectives, applied to dependency grammar induction. In Proceedings of EMNLP. find
  46. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL. find
  47. Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL. find
  48. Stephen Tratz and Eduard Hovy. 2011. A fast, effective, non-projective, semantically-enriched parser. In Proceedings of EMNLP. find
  49. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL. find our card
  50. Jakob Uszkoreit and Thorsten Brants. 2008. Distributed word clustering for large scale class-based language modeling in machine translation. In Proceedings of ACL-HLT. find
  51. Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 11(02):207–238. find
  52. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In IJC-NLP Workshop: NLP for Less Privileged Languages. find
  53. Yue Zhang and Stephen Clark. 2008. A tale of two parsers: Investigating and combining graph-based and transition-based dependency parsing. In Proceedings of EMNLP. find our card
  54. Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of ACL-HLT. find