Elamur Research⌕
☰
home / papers / 2019 / Learning and Evaluating General Linguistic Intelligence

Learning and Evaluating General Linguistic Intelligence

Paper · 2019 · Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor et al.

Summary

The authors argue that standard leaderboard scores overstate progress in NLP. They propose a notion of "general linguistic intelligence": a system's capacity to reuse what it already knows about a language's words, grammar, meaning and usage so it can master unfamiliar tasks from few examples. To measure this, they add a metric based on online (prequential) coding, following Blier & Ollivier (2018). It scores how many task-specific examples a model needs before it performs well, rather than only its final accuracy.

They test pretrained BERT and an ELMo-based BiDAF model on several reading-comprehension, semantic role labeling, relation extraction and inference datasets, under four setups:

  • fine-tuning,
  • transfer between related tasks,
  • cross-dataset generalization,
  • continual learning.

Both models still need substantial in-domain data and gain clearly from related-task pretraining. However, without fine-tuning they transfer poorly between datasets of the same task, which suggests overfitting to quirks of particular datasets such as SQuAD. They also forget earlier tasks when trained sequentially. The study matters because it reframes evaluation around sample efficiency and robustness. It also points to remedies such as memory modules, meta-learning and better curricula.

Key points

  • Models compared:
    • BERT base: 12 Transformer layers, 12 attention heads, 768 hidden units, about 110M parameters.
    • ELMo: about 100M parameters, with a 300-dimensional BiLSTM and a BiDAF network on top.
  • Datasets: SQuAD 1.1, TriviaQA (Web section), QuAC, QA-SRL, the zero-shot relation extraction set (QA-ZRE), MNLI and SNLI. Most tasks are cast as span prediction.
  • New metric: an online codelength measure that captures how fast a model adapts, rewarding sample efficiency on top of conventional end-of-training scores.
  • Training order matters: sequential (continual) training causes catastrophic forgetting. In contrast, a random curriculum over 200,000 iterations lets BERT keep competence across all tasks, with these results:
ModelSQuADMNLISNLIQA-ZRE
BERT85.481.188.088.2
ELMo78.369.177.988.5
  • Proposed directions: elastic weight consolidation, progress and compress, neural cache and dynamic evaluation, meta-learning (with the codelength as a possible objective), and Population Based Training for designing curricula.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer and Phil Blunsom. 2019. Learning and Evaluating General Linguistic Intelligence. arXiv:1901.11373.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1901.11373 · arXiv 1901.11373

@misc{yogatama2019learning,
  title = {{Learning and Evaluating General Linguistic Intelligence}},
  author = {Dani Yogatama, Cyprien {de Masson d'Autume}, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer and Phil Blunsom},
  year = {2019},
  doi = {10.48550/arxiv.1901.11373},
  eprint = {1901.11373},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.11373}
}

References

36 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Kuttler, Andrew Lefrancq, Simon Green, Víctor Valdes, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and… — DeepMind Lab
  2. Leonard Blier and Yann Ollivier. The description length of deep learning models. In Proc. of NIPS, 2018. find
  3. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proc. of EMNLP, 2015.
  4. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint, 2016. find
  5. Gregory J. Chaitin. On the intelligibility of the universe and the notions of simplicity, complexity and irreducibility. In Thinking about Godel and Turing: Essays on Complexity, 1970–2007. find
  6. World Scientific, 2007. find
  7. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proc. of EMNLP, 2018.
  8. Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In Proc. of NIPS, 2015. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint, 2018.
  10. Tim Van Erven, Peter Grunwald, and Steven De Rooij. Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC-BIC dilemma. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(3):361––417, 2012. find
  11. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. of ICML, 2017.
  12. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. Large-scale QA-SRL parsing. In Proc. of ACL, 2018.
  13. Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. find
  14. Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In Proc. of ICLR, 2017.
  15. Isabelle Guyon, Gavin Cawley, Gideon Dror, and Vincent Lemaire. Results of the active learning challenge. In Proc. of Workshop on Active Learning and Experimental Design, 2011. find
  16. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… — Achieving Human Parity on Automatic Chinese to English News Translation
  17. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. find
  18. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv preprint, 2017. find
  19. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proc. of EMNLP, 2017.
  20. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL, 2017.
  21. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc… find
  22. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of sequence models. In Proc. of ICML, 2018. find
  23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Quest… — Natural Questions: A Benchmark for Question Answering Research
  24. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proc. of CoNLL, 2017.
  25. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint, 2018.
  26. Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24(92):109–165, 1989. find
  27. Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint, 2018. find
  28. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
  29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. arXiv preprint, 2018. find
  30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP, 2016.
  31. Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress and compress: A scalable framework for continual learning. In Proc. of ICML, 2018. find
  32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. Bi-directional attention flow for machine comprehension. In Proc. of ICLR, 2017. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NIPS, 2017.
  34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint, 2018.
  35. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL-HLT, 2018.
  36. J. Gerard Wolff. Language acquisition, data compression and generalization. Language & Communication, 2(1):57–89, 1982. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yogatama-2019-learning-evaluating-general-linguistic-intelligence/