Elamur Research⌕
☰
home / papers / 2020 / SyntaxGym: An Online Platform for Targeted Evaluation of Language Models

SyntaxGym: An Online Platform for Targeted Evaluation of Language Models

Summary

Targeted syntactic evaluation tests whether neural language models have learned specific grammatical generalizations. Broad metrics such as perplexity can hide large differences in syntactic knowledge between models that score similarly. This kind of work, however, requires both psycholinguistic design expertise and the engineering capacity to run large models. The authors close this gap with SyntaxGym, a website (syntaxgym.org) backed by open-source tooling.

Linguists contribute test suites written in a standardized JSON format. NLP researchers contribute models packaged as Docker images that follow a shared API. The backend validates submissions, scores every model against every suite in the cloud, stores raw surprisals and accuracies, and presents interactive visualizations. Two command-line tools, syntaxgym and lm-zoo, let anyone reproduce these evaluations on their own machine.

Compared with BLiMP, the framework supports stricter predictions, such as conjunctions of inequalities across conditions. It also measures surprisal only within pre-specified critical regions. The contribution is infrastructure rather than a new empirical finding: it makes targeted evaluation more standardized, reproducible and accessible to both communities.

Key points

  • Test suite components: suites are built from regions, conditions, items and predictions.
    • Predictions are inequalities over model surprisal within chosen regions, for example the matrix verb in subject–verb agreement items.
    • A model's accuracy is the share of items for which the prediction holds.
  • lm-zoo API: each model is wrapped in a Docker image exposing three standard commands: tokenize, get-surprisals and unkify. Documentation is at cpllab.github.io/lm-zoo.
  • syntaxgym CLI: it lists hosted models and suites and evaluates either hosted or user-supplied ones.
    • Example models: gpt-2, gpt-2-xl, transformer-xl.
    • Example suites: number-orc, number-src, mvrr.
    • In a demo run, gpt-2 scores 0.7857 (22/28) on mvrr.
  • Related datasets: the paper positions itself against Marvin and Linzen (2018) and BLiMP (Warstadt et al., 2020).
    • Marvin and Linzen covers few phenomena.
    • BLiMP sentences are template-generated and complementary to SyntaxGym's hand-designed suites.
  • Planned extensions: human performance as a reference metric, and broader lm-zoo features such as next-word distributions and internal representations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

85 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-demos.10

@inproceedings{gauthier2020syntaxgym,
  title = {{SyntaxGym: An Online Platform for Targeted Evaluation of Language Models}},
  author = {Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.acl-demos.10},
  url = {https://doi.org/10.18653/v1/2020.acl-demos.10}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn’t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… find
  2. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. find
  3. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations. find
  4. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In Proceedings of NAACL-HLT, pages 199–209. find
  5. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… find our card
  6. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics. find
  7. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  8. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL-HLT, pages 1195–1205. find our card
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  10. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Association of Computational Linguistics. find
  11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  12. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 5:1202–1247. find
  13. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In Transactions of the Association for Computational Linguistics, volume 4, pages 521–535. find our card
  14. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/