Elamur Research⌕
☰
home / papers / 2021 / Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?

Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?

Summary

The authors test whether recurrent networks actually acquire syntactic structure when they learn subject–verb number agreement in English. Their hypothesis is as follows: a model trained only on "hard" sentences, where at least one attractor noun of the opposite number stands between the subject and the verb, should also have no trouble with agreement in simple sentences without attractors. That is, provided it has acquired the grammatical rule. The experiments cover four architectures (LSTM, GRU, ONLSTM, Decay RNN) and two training regimes. The first is language modeling. The second is binary grammaticality classification on data augmented with counterfactual examples. The data are taken from a corpus of Wikipedia sentences (Linzen et al., 2016) and the synthetic dataset of Marvin & Linzen (2018). After such training, the models handle simple sentences noticeably worse than complex ones. Even ONLSTM, with its soft hierarchical bias, does not fix this. In the authors' view, high scores on the agreement task may mean that a model has adapted to statistical regularities rather than acquired the grammar. Models should therefore be evaluated on data of varying type and difficulty.

Key points

  • Two training sets are compared: a natural one, where the number of attractors is unrestricted, and a selective one, where every sentence contains at least one attractor. The test set contains 157 thousand corpus sentences; each model was trained with 5 different seeds.
  • LSTM as an LM achieves an accuracy of 0.98 on sentences without attractors after natural training and only 0.89 after selective training. Among the classifiers, the drop after selective training is even larger: LSTM 0.60, GRU 0.62, ONLSTM 0.64, Decay RNN 0.70. On sentences with attractors, the same classifiers score 0.94–0.98.
  • On synthetic sentences, the LSTM LM after selective training scores 0.63 without attractors and 0.83 with one attractor.
  • In the selective set, over 30% of intervening nouns have the same number as the subject, i.e., they are not attractors. Hence the result cannot be explained by a primitive rule such as "do not agree with the nearest noun." The models learn more complex heuristics, but these are still superficial.
  • The authors compared the models' hidden representations using RSA on 2000 sentences. The representations cluster by training set rather than by architecture. In the classifiers, the representations depend more strongly on the seed, so the authors recommend the LM objective for comparing models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2010.04976

@inproceedings{bansal2021can,
  title = {{Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?}},
  author = {Bansal, Hritik and Bhatt, Gantavya and Agarwal, Sumeet},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2021},
  year = {2021},
  eprint = {2010.04976},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.04976}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gantavya Bhatt, Hritik Bansal, Rishubh Singh, and Sumeet Agarwal. 2020. How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 244– 254, Onl… find
  2. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  3. J Kathryn Bock and Carol A Miller. 1991. Broken agreement. Cognitive psychology, 23(1):45–93. find
  4. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2093—2098, Austin, TX. find
  5. Rui Chaves. 2020. What don’t RNN language models learn about filler-gap dependencies? In Proceedings of the Society for Computation in Linguistics 2020, pages 1–11, New York, New York. Association for Computational Linguistics. find
  6. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140. find our card
  7. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu… find
  8. Hiroshi Noji and Hiroya Takamura. 2020. An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3375–3385, Online. Association for Computat… find
  9. Yiding Hao. 2020. Attribution analysis of grammatical dependencies in lstms. arXiv preprint arXiv:2005.00062. — [arXiv:2005.00062](https://arxiv.org/abs/2005.00062) find
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  11. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. find
  12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings. find
  13. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. arXiv preprint arXiv:1906.06438. find
  14. Aarre Laakso and Garrison Cottrell. 2000. Content and cluster analysis: assessing representational similarity in neural systems. Philosophical psychology, 13(1):47–76. find
  15. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  16. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… find our card
  17. Luzi Sennhauser and Robert C Berwick. 2018. Evaluating the ability of lstms to learn context-free grammars. arXiv preprint arXiv:1811.02611. find
  18. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bansal-2021-can-rnns-trained-harder-subject/