Elamur Research⌕
☰
home / papers / 2021 / Targeting the Benchmark: On Methodology in Current Natural Language Processing Research

Targeting the Benchmark: On Methodology in Current Natural Language Processing Research

Paper · 2021 · David Schlangen

Summary

This position paper questions a familiar NLP routine. A group proposes a language task with a dataset and a baseline. Others improve the scores, and the field moves on, usually without saying what this progress achieves. Schlangen offers no new data or models. Instead, he breaks the benchmark-driven method into parts: cognitive capabilities, the tasks thought to involve them, the datasets that exemplify those tasks, and the models built on the datasets. He then asks what each link can actually support.

He argues that NLP benchmarks differ from classic computing benchmarks in two ways. They serve as training material as well as a measuring tool, and all of them sample one underlying phenomenon, language use. A task is therefore valuable only if it points beyond itself to clearly specified, falsifiable capabilities. Models trained on it should also carry over to related tasks.

His diagnosis is that the field optimises individual tasks well but rarely examines transfer between them. He recommends renewed contact with linguistics and psychology, plus a structured curriculum of tasks.

Key points

  • Figure 1, adapted from Schlangen (2019), connects capabilities, tasks, datasets and models. It shows that links between different tasks and models are seldom made explicit.
  • The paper separates three kinds of task by their purpose: product tasks, annotation tasks, and benchmark tasks.
  • It distinguishes verifying a dataset (checking annotation correctness) from validating it (arguing that it represents the task). Visual question answering is the worked example. The dataset of Antol et al. (2015) could be handled without images, a "language bias" noted by Jabri et al. (2016). Goyal et al. (2017) then built a less biased dataset.
  • Following Popper, capability claims are more valuable the more precisely and falsifiably they are stated. Evaluating pre-trained models across several tasks at once can reveal capabilities the tasks have in common, as shown in the 2018 work of Devlin et al. and Peters et al. and in Wang et al. (2019a,b).
  • The proposal is to arrange tasks in a hierarchy of complexity and inclusion. The benchmarking target would then be a model's developmental trajectory through it. The paper appeared in ACL-IJCNLP 2021 Short Papers, pp. 670–674.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PotsdamPotsdam · Germany46 works · 5,393 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as David Schlangen. 2021. Targeting the Benchmark: On Methodology in Current Natural Language Processing Research. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.85 · arXiv 2007.04792

@inproceedings{schlangen2021targeting,
  title = {{Targeting the Benchmark: On Methodology in Current Natural Language Processing Research}},
  author = {David Schlangen},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.85},
  eprint = {2007.04792},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-short.85}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV). find
  2. Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistic, 34(4):555–596. find
  3. Emily M Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2, pages 5185–5198.
  4. Denny Borsboom. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge University Press, Cambridge, UK. find
  5. Andrew Butterfield, Gerard Ekembe Ngondi, and Anne Kerr, editors. 2016. A Dictionary of Computer Science, 7th edition. Oxford University Press, Oxford, UK. find
  6. François Chollet. 2019. On the Measure of Intelligence. arXiv e-prints, page arXiv:1911.01547. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  8. Dheeru Dua and Casey Graff. 2019. UCI machine learning repository. find
  9. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In CVPR 2017.
  10. Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016. Revisiting Visual Question Answering Baselines. In European Conference on Computer Vision (ECCV). find
  11. Fernando Martinez-Plumed and José Hernandez-Orallo. 2018. Dual indicators to analyse ai benchmarks: Difficulty, discrimination, ability and generality. IEEE Transactions on Games, pages 1–1. find
  12. Emiel van Miltenburg. 2019. Pragmatic factors in (automatic) image description. Ph.D. thesis, Vrije Universiteit Amsterdam. find
  13. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694.
  14. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  15. Karl Popper. 1934. Logik der Forschung. Siebeck, Tübingen, Germany. find
  16. Mohr James Pustejovsky and Amber Stubbs. 2013. Natural Language Annotation for Machine Learning. O’Reilly, Sebastopol, CA, USA. find
  17. David Schlangen. 2019. Language tasks and language games: On methodology in current natural language processing research. CoRR, abs/1908.10747. — [arXiv:1908.10747](https://arxiv.org/abs/1908.10747) find
  18. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Super-GLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS, July, pages 1–30. find
  19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR 2019, pages 1–20.
  20. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  21. Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2018. find
  22. Taskonomy: Disentangling Task Transfer Learning. In CVPR 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlangen-2021-targeting-benchmark-methodology-current-natural/