Elamur Research⌕
☰
home / papers / 2017 / Fine-Grained Prediction of Syntactic Typology: Discovering Latent Structure with Supervised Learning

Fine-Grained Prediction of Syntactic Typology: Discovering Latent Structure with Supervised Learning

Paper · 2017 · Dingquan Wang and Jason Eisner

Summary

The paper asks whether the basic word-order profile of an unseen language can be inferred from only a corpus of POS tag sequences. For each of 57 UD dependency relations, the target is a directionality score in [0, 1]: the share of tokens where the dependent appears to the right of its head (e.g., how often dobj follows its verb). The authors do not treat this as unsupervised grammar induction. Instead, they frame it as supervised prediction in which each training example is an entire language. Real treebanks are too few for this, so they add synthetic languages from Galactic Dependencies, built by reordering dependents in UD treebanks to imitate other languages' word orders. The models use hand-engineered or neural features of POS sequences, and a simple expected-count heuristic serves as a baseline. Adding synthetic languages to a small set of real ones improves accuracy. The system also tolerates noisy tags and clearly beats a grammar-induction baseline. The predicted vectors could serve as language embeddings or as priors for parsing and grammar induction.

Key points

  • Data: UD version 1.2 (treebanks for 37 real languages) and GD (Galactic Dependencies) version 1.0 (53,428 synthetic, projective treebanks in the UD format).
  • Evaluation uses an ε-insensitive L1-style loss with ε = 0.1, weighted by how frequent each relation is in the language, and is computed on held-out real languages.
  • Baseline "expected count" (EC): nearby tag pairs in the test corpus cast soft votes for left or right attachment, based on how often such pairs are linked in the training treebanks.
  • Recurrent neural features did not clearly outperform hand-engineered ones. The authors suggest RNNs may help in future work that uses raw words with word embeddings.
  • The authors believe this is the first use of synthetic languages to train a learner that is applied to real languages. They contrast it with WALS-based resources and with earlier, narrower predictors such as Saha Roy et al. (2014) and Zhang et al. (2016).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.Universal Dependencies · 184 works

Cite

Cite as Dingquan Wang and Jason Eisner. 2017. Fine-Grained Prediction of Syntactic Typology: Discovering Latent Structure with Supervised Learning. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00052

@article{wang2017fine,
  title = {{Fine-Grained Prediction of Syntactic Typology: Discovering Latent Structure with Supervised Learning}},
  author = {Dingquan Wang and Jason Eisner},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00052},
  url = {https://doi.org/10.1162/tacl_a_00052}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew S. Dryer and Martin Haspelmath, editors. The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig, 2013. http://wals.info/. ↗
  2. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. Many languages, one parser. Transactions of the Association of Computational Linguistics, 4:431–444, 2016.
  3. Emily M. Bender. Linguistically naı̈ve != language independent: Why NLP needs linguistic typology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pages 26–32, 2009. find
  4. Glenn Carroll and Eugene Charniak. Two experiments on learning probabilistic dependency grammars from corpora. In Working Notes of the AAAI Workshop on Statistically-Based NLP Techniques, pages 1–13, 1992. find
  5. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, 2014.
  6. Noam Chomsky. Lectures on Government and Binding: The Pisa Lectures. Holland: Foris Publications, 1981. find
  7. Noam Chomsky and Howard Lasnik. The theory of principles and parameters. In Syntax: An International Handbook of Contemporary Research, Joachim Jacobs, Arnim von Stechow, Wolfgang Sternefeld, and Theo Vennemann, editors. Berlin: de Gruyter, 1993. find
  8. William Croft. Typology and Universals. Cambridge University Press, 2002. find
  9. Hal Daumé III and Lyle Campbell. A Bayesian model for discovering typological implications. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 65–72, 2007. find
  10. Joakim Nivre, et al. Universal Dependencies 1.2. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University in Prague. Data available at http://universaldependencies.org, 2015. ↗
  11. Robert Frank and Shyam Kapur. On the use of triggers in parameter setting. Linguistic Inquiry, 27: 623–660, 1996. find
  12. Richard Futrell, Kyle Mahowald, and Edward Gibson. Quantifying word order freedom in dependency corpora. In Proceedings of the Third International Conference on Dependency Linguistics, pages 91–100, 2015. find
  13. Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11:2001–2049, 2010. find
  14. Ryan Georgi, Fei Xia, and William Lewis. Comparing language similarity across genetic and typologically-based groupings. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 385–393, 2010. find
  15. Edward Gibson and Kenneth Wexler. Triggers. Linguistic Inquiry, 25(3):407–454, 1994. find
  16. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2010. find
  17. Joseph H. Greenberg. Some universals of grammar with particular reference to the order of meaningful elements. In Universals of Language, Joseph H. Greenberg, editor, pages 73–113. MIT Press, 1963. find
  18. John A. Hawkins. Word Order Universals. Elsevier, 2014. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. find
  20. Dan Klein and Christopher Manning. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 478–485, 2004. find
  21. Karim Lari and Steve J. Young. The estimation of stochastic context-free grammars using the Inside-Outside algorithm. Computer Speech and Language, 4(1):35–56, 1990. find
  22. Phong Le and Willem Zuidema. Unsupervised dependency parsing: Let’s use supervised parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 651–661, 2015. find
  23. William D. Lewis and Fei Xia. Automatically identifying computationally relevant typological features. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II, 2008. find
  24. Haitao Liu. Dependency direction as a means of word-order typology: A method based on dependency treebanks. Lingua, 120(6):1567–1578, 2010. find
  25. David Mareček and Milan Straka. Stop-probability estimates computed on a large corpus improve unsupervised dependency parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 281–290, 2013. find
  26. Yugo Murawaki. Continuous space representations of linguistic typology and their application to phylogenetic inference. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 324–334, 2015. find
  27. Tahira Naseem, Harr Chen, Regina Barzilay, and Mark Johnson. Using universal linguistic knowledge to guide grammar induction. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1234–1244, 2010.
  28. Tahira Naseem, Regina Barzilay, and Amir Globerson. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 629–637, 2012.
  29. Dat Quoc Nguyen, Dai Quoc Nguyen, Dang Duc Pham, and Son Bao Pham. RDRPOSTagger: A ripple down rules-based part-of-speech tagger. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 17– 20, 2014. find
  30. Hiroshi Noji, Yusuke Miyao, and Mark Johnson. Using left-corner parsing to encode universal structural constraints in grammar induction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 33–43, 2016. find
  31. Helen O’Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, and Anna Korhonen. Survey on the use of typological information in natural language processing. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, pages 1297–1308, 2016. find
  32. Rishiraj Saha Roy, Rahul Katare, Niloy Ganguly, and Monojit Choudhury. Automatic discovery of adposition typology. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers, pages 1037–1046, 2014. find
  33. Jae Jung Song. Linguistic Typology: Morphology and Syntax. Routledge, 2014. find
  34. Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 751–759, 20… find
  35. Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. Breaking out of local optima with count transforms and model recombination: A study in grammar induction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1983–1995, 2013. find
  36. Oscar Täckström, Ryan McDonald, and Joakim Nivre. Target language adaptation of discriminative transfer parsers. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1061–1071, 2013.
  37. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012. find
  38. Dingquan Wang and Jason Eisner. The Galactic Dependencies treebanks: Getting more data by synthesizing new languages. Transactions of the Association of Computational Linguistics, 4: 491–505, 2016. Data available at https://github.com/gdtreebank/gdtreebank. ↗
  39. Yuan Zhang and Regina Barzilay. Hierarchical low-rank tensors for multilingual transfer parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1857–1867, 2015.
  40. Yuan Zhang, Roi Reichart, Regina Barzilay, and Amir Globerson. Learning to map into a universal POS tagset. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1368–1378, 2012. find
  41. Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. Ten pairs to tag—multilingual POS tagging via coarse mapping between embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1307… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2017-fine-grained-prediction-syntactic-typology/