Elamur Research⌕
☰
home / papers / 2020 / A closer look at the performance of neural language models on reflexive anaphor licensing

A closer look at the performance of neural language models on reflexive anaphor licensing

Summary

Earlier studies (Marvin and Linzen, 2018; Futrell et al., 2018) concluded that neural language models (NLMs) fail to learn English reflexive anaphor licensing (RAL), meaning the rules that decide which noun phrase can serve as the antecedent of words like himself or themselves. Hu, Chen and Levy argue that those conclusions were partly caused by flaws in experimental design. One flaw was comparing different reflexives in the same context, even though themselves, himself and herself have very different corpus frequencies. Another was relying on profession nouns that are only stereotypically gendered and too rare for small-vocabulary models.

The authors built new test suites in which the target reflexive stays fixed and only the preceding context changes. Each item appears in three conditions: Baseline, Distractor and Ungrammatical. They test number agreement and rely on frequent nouns whose gender is built into the word itself, for example man and woman. The experiments separately probe the two licensing constraints, LOCALITY and C-COMMAND.

Under this paradigm, models show clearly more evidence of learning RAL than previously reported. The work shows that careful psycholinguistic controls are needed to fairly assess what NLMs can learn.

Key points

  • The evaluation counts an item as correct only when the target reflexive is least probable in the Ungrammatical condition, compared with both Distractor and Baseline. This sets chance at 33.33% rather than the usual 50%.
  • Models tested include large-vocabulary models (BERT, TransXL, JRNN, GRNN) and small-vocabulary models (RNNG, TinyLSTM).
  • In sentential complement items, all large-vocabulary models score near ceiling. RNNG beats TinyLSTM, though both score lower than the large models.
  • A new experiment isolates C-COMMAND by placing a distractor inside a PP modifier within the same clause. Here all neural models except TinyLSTM perform well above chance, with RNNG at 62% versus 43% for TinyLSTM.
  • With the older profession-noun materials, 11 of 12 nouns are stereotypically male. This produces a himself/herself asymmetry that largely disappears with lexically gendered nouns. Code and data are released on GitHub (jennhu/reflexive-anaphor-licensing).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hu2020closer,
  title = {{A closer look at the performance of neural language models on reflexive anaphor licensing}},
  author = {Hu, Jennifer and Chen, Sherry Yong and Levy, Roger},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2020},
  year = {2020},
  doi = {10.7275/67qw-mf84},
  url = {https://doi.org/10.7275/67qw-mf84}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Veronica Boyce, Titus von der Malsburg, Till Poppels, and Roger Levy. 2019. Remember ‘him’, forget ‘her’: Gender bias in the comprehension of prono-mial referents. In 32nd Annual CUNY Conference on Human Sentence Processing. find
  2. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. Technical report, Google. find our card
  3. Noam Chomsky. 1993. Lectures on government and binding: The Pisa lectures. 9. Walter de Gruyter. find
  4. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. CoRR, abs/1602.07776. find
  7. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  8. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. CoRR, abs/1809.01329. find
  9. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. find
  10. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. CoRR, abs/1901.05287. find our card
  11. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find our card
  12. John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan R. Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), pages 2727–2736, Melbourne, Australia. Association for Computati… find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  14. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  15. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers… find
  16. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 5:1202–1247. find
  17. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In Transactions of the Association for Computational Linguistics, volume 4, pages 521–535. find our card
  18. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  19. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  20. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Proceedings of ICLR. find our card
  21. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association, pages 1045–1048, Makuhari, Chiba, Japan. find
  22. Tanya Reinhart. 1981. Definite NP anaphora and c-command domains. Linguistic Inquiry, 12(4):605– 635. find
  23. Tanya Reinhart. 1983. Anaphora and semantic interpretation. Routledge. find
  24. Luigi Rizzi. 2013. Locality. Lingua, 130:169–186. find
  25. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 8–14, New Orleans, Louisiana… find our card
  26. Mitchell Stern, Daniel Fried, and Dan Klein. 2017. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700. Association for Computational Linguistics. find
  27. Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association. find
  28. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736, Brussels, Belgium. Association for Computational Linguistics. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  30. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about fillergap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computatio… find
  31. Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballestros, and Roger Levy. 2019. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  32. Thomas Wolf. 2019. Some additional experiments extending the tech report “Assessing BERT’s syntactic abilities” by Yoav Goldberg. Technical report, HuggingFace, Inc. find our card
  33. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pa… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/hu-2020-closer-look-performance-neural-language/