Elamur Research
☰
home / papers / 2018 / Colorless Green Recurrent Networks Dream Hierarchically

Colorless Green Recurrent Networks Dream Hierarchically

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1108.pdf

Summary

The paper asks whether recurrent networks trained only for next-word prediction learn abstract syntactic structure, or whether they mostly rely on surface patterns. Earlier work by Linzen et al. (2016) suggested that LSTMs need explicit supervision to handle long-distance agreement. The authors revisit that claim. They automatically extract number-agreement constructions from dependency treebanks in Italian, English, Hebrew and Russian. They then turn each test sentence into a "nonce" version by swapping every content word for a random word with the same morphology. The result stays grammatical but has no meaning, so the model can't lean on semantic or collocational cues. The language-model-trained LSTMs predict the correct singular or plural form of the target word reliably on both original and nonsense sentences. In Italian their accuracy comes close to that of human annotators. Models with lower perplexity were also better at agreement. This suggests that a plain LM objective can produce real hierarchical generalizations, and that careful tuning matters.

Key points

  • The benchmark has cue–target pairs linked by a dependency arc, with at least 3 intervening tokens and at least 10 singular and 10 plural instances. This gave between 2 (English) and 21 (Russian) constructions per language. VP coordination was the only construction found in all four languages.
  • The average distance between cue and target ranged from 3.6 words (Hebrew) to 4.5 words (Italian). The constructions are rare: even in Hebrew they appear in only about 0.8% of sentences.
  • For Italian, the authors collected human judgments on both original and nonce sentences. They describe this as the first such dataset for nonsensical agreement.
  • On the Linzen et al. (2016) subject–verb test set, with 0–4 attractors and 187 sentences removed because they overlapped with training data, the tuned LSTM LM clearly beat their "Google LM". It came close to their supervised model.
  • The code, data and Italian human judgments are released at github.com/facebookresearch/colorlessgreenRNNs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

294 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 271 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1108 · arXiv 1803.11138

@inproceedings{gulordava2018colorless,
  title = {{Colorless Green Recurrent Networks Dream Hierarchically}},
  author = {Gulordava, Kristina and Bojanowski, Piotr and Grave, Edouard and Linzen, Tal and Baroni, Marco},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1108},
  eprint = {1803.11138},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1108}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track. Toulon, France. Published online: https://openreview.net/group? id=ICLR.cc/2017/conference. ↗ our card
  2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of ACL. Vancouver, Canada, pages 861–872. find our card
  3. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology 15(2):1–15. find
  4. Barry Blake. 2001. Case. MIT Press, Cambridge, MA. find
  5. Kathryn Bock and Carol Miller. 1991. Broken agreement. Cognitive Psychology 23(1):45–93. find
  6. Bo Cartling. 2008. On the implicit acquisition of a context-free grammar by a simple recurrent neural network. Neurocomputing 71:1527–1537. find
  7. Ciprian Chelba, Mohammad Norouzi, and Samy Bengio. 2017. N-gram language modeling using recurrent neural network estimation. arXiv preprint arXiv:1703.10724 . — [arXiv:1703.10724](https://arxiv.org/abs/1703.10724) find
  8. Noam Chomsky. 1957. Syntactic Structures. Mouton, Berlin, Germany. find
  9. Morten Christiansen and Nick Chater. 1999. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23(2):157–205. find
  10. James Cross and Liang Huang. 2016. Incremental parsing with minimal features using bi-directional LSTM. In Proceedings of ACL (Short Papers). Berlin, Germany, pages 32–37. find
  11. Peter Culicover and Paul Postal, editors. 2001. Parasitic gaps. MIT Press, Cambridge, MA. find
  12. Jeffrey Elman. 1990. Finding structure in time. Cognitive Science 14:179–211. find
  13. Jeffrey Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7:195–225. find
  14. Jeffrey Elman. 1993. Learning and development in neural networks: The importance of starting small. Cognition 48:71–99. find
  15. Émile Enguehard, Yoav Goldberg, and Tal Linzen. 2017. Exploring the syntactic abilities of RNNs with multi-task learning. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). pages 3–14. find
  16. Martin Everaert, Marinus Huybregts, Noam Chomsky, Robert Berwick, and Johan Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cognitive Sciences 19(12):729– 743. find
  17. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. Irstlm: An open source toolkit for handling large scale language models. In Ninth Annual Conference of the International Speech Communication Association. pages 1618–1621. find
  18. Angela D. Friederici, Martin Meyer, and D. Yves Von Cramon. 2000. Auditory language comprehension: An event-related fMRI study on the processing of syntactic and lexical information. Brain and Language 74(2):289–300. find
  19. Felix Gers and Jürgen Schmidhuber. 2001. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks 12(6):1333–1340. find
  20. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks. Springer, Berlin. find
  21. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 187– 197. find our card
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–178–. find
  23. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2017. Visualisation and diagnostic classifiers reveal how recurrent and recursive neural networks process hierarchical structure. http:// arxiv.org/abs/1711.10203. — [arXiv:1711.10203](https://arxiv.org/abs/1711.10203) find
  24. Matt A. Johnson and Adele E. Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes 28(10):1439–1452. find
  25. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics 4:313–327. find our card
  26. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., 1995 International Conference on. IEEE, volume 1, pages 181–184. find
  27. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4:521– 535. find our card
  28. Heidi Lorimor, Kathryn Bock, Ekaterina Zalkind, Alina Sheyman, and Robert Beard. 2008. Agreement and attraction in Russian. Language and Cognitive Processes 23(6):769–799. find
  29. Tomas Mikolov. 2012. Statistical language models based on neural networks. Dissertation, Brno University of Technology. find
  30. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH. Makuhari, Japan, pages 1045–1048. find
  31. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of NAACL. Atlanta, Georgia, pages 746–751. find
  32. Andrea Moro, Marco Tettamanti, Daniela Perani, Caterina Donati, Stefano F Cappa, and Ferruccio Fazio. 2001. Syntax and the brain: disentangling grammar by selective anomalies. Neuroimage 13(1):110–118. find
  33. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari (Confer… find our card
  34. Fernando Pereira. 2000. Formal grammar and information theory: together again? Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 358(1769):1239–1253. find
  35. Douglas Rohde and David Plaut. 1997. Simple recurrent networks and natural language: How important is starting small? In Proceedings of CogSci. Stanford, CA, pages 656–661. find
  36. Helmut Schmid. 1995. Improvements in part-of-speech tagging with an application to German. In Proceedings of the EACL-SIGDAT Workshop. Dublin, Ireland. find
  37. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing MT quality with contrastive translation pairs. In Proceedings of EACL (Short Papers). Valencia, Spain, pages 376–382. find our card
  38. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of EMNLP. Austin, Texas, pages 1526– 1534. find our card
  39. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nish… find
  40. Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. In International Conference on Learning Representations. https:// openreview.net/forum?id=SkFqf0lAZ. find
  41. Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017. Dependency parsing as head selection. In Proceedings of EACL. Valencia, Spain, pages 665– 676. find