Elamur Research⌕
☰
home / papers / 2020 / The Turking Test: Can Language Models Understand Instructions?

The Turking Test: Can Language Models Understand Instructions?

Paper · 2020 · Avia Efrat and Omer Levy

Summary

Efrat and Levy ask whether a pretrained language model can perform a new task when it receives only a natural-language description of the task instead of labeled examples. They propose the Turking Test, a set of instruction understanding tasks (IUTs) ranging from demanding to trivial. The hardest tasks have the model act as a crowdworker and create new examples for SNLI, SQuAD and NewsQA. A middle tier asks it to list the nouns in a sentence, optionally under a condition. The simplest tasks ask the model to return the word or character found at a given position n within a sentence taken from GLUE's diagnostic set.

GPT-2 (1.5B parameters, Hugging Face implementation) was evaluated under deliberately generous conditions. Instructions were tuned on a few resources, extra text before or after a valid answer was forgiven, and the better of greedy or nucleus decoding was reported. Even so, the model failed almost everywhere, frequently disregarding explicit constraints or just echoing its input. The authors argue that instructions strictly generalize few-shot prompting because they can also specify what to avoid. This makes instruction following a valuable but still unsolved goal.

Key points

  • Turking tasks: each task produced 500 outputs (100 resources × 5 nucleus samples, p = 0.9), judged by 10 annotators with 98% observed agreement. Accuracy was 0.4% on SNLI, 0.2% on SQuAD and 4.2% on NewsQA. Format-consistency rates were 12%, 6% and 22%. Greedy decoding scored 0%.
  • Prompt format mattered: replacing the original HTML-based visual cues with plain textual prompts was necessary. Without that change, accuracy was 0% on every turking task.
  • Noun listing: adding a simple condition dropped accuracy to 3.5%. Greedy outputs repeated the resource or the instructions in 10% of unconditioned outputs and 30% of conditioned ones.
  • Index retrieval: the test used 832 GLUE diagnostic sentences (1,664 outputs per task). Accuracy was 2.0% for the nth word and 1.3% for the nth character. Both fall below trivial baselines of always answering "the" (6.4%) or "e" (13.8%).
  • Repetition and sensitivity: repetition appeared in 46% of nth-word outputs and 59% of nth-character outputs. Repetition patterns differed sharply between these two near-identical prompts (96% of character outputs began with the resource vs. none of the word outputs). This suggests the model is hypersensitive to small changes in wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tel Aviv UniversityTel Aviv · Israel89 works · 11,326 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Avia Efrat and Omer Levy. 2020. The Turking Test: Can Language Models Understand Instructions?. arXiv:2010.11982.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.11982 · arXiv 2010.11982

@misc{efrat2020turking,
  title = {{The Turking Test: Can Language Models Understand Instructions?}},
  author = {Avia Efrat and Omer Levy},
  year = {2020},
  doi = {10.48550/arxiv.2010.11982},
  eprint = {2010.11982},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.11982}
}

References

11 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. License…
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  5. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342–1352, Vancouver, Canada. Association for Computational…
  6. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. CoRR, abs/1904.09751.
  7. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  8. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250. Licensed under CC BY-SA 4.0.
  9. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. CoRR, abs/1611.09830.
  10. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461.
  11. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… — Transformers: State-of-the-Art Natural Language Processing

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/efrat-2020-turking-test-language-models-understand/