Elamur Research⌕
☰
home / papers / 2020 / How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context

How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context

Paper · 2020 · Jey Han Lau, Carlos Armendariz, Shalom Lappin et al.

Summary

The paper examines how surrounding text shapes human judgments of sentence naturalness, and whether unsupervised language models can predict those judgments. The authors took 50 English Wikipedia sentences and created degraded variants with round-trip machine translation using Moses. They then collected crowdsourced ratings on Amazon Mechanical Turk under three conditions: the sentence alone, the sentence after its real three-sentence preceding context, and the sentence after an unrelated passage from another article. A topic-identification step ensured that workers actually read the context.

Context of either kind compressed the ratings: weak sentences scored higher and strong ones lower. The authors attribute this to cognitive load. Real context also raised scores across the board compared with random context, which points to a separate discourse coherence effect.

In the modeling experiments, bidirectional transformers such as BERT clearly beat left-to-right recurrent models. The strongest model reached roughly human-level agreement, setting a new state of the art for unsupervised acceptability prediction. This questions whether left-to-right processing is the right bias for these models.

Key points

  • Test set: 250 sentences, made up of 50 originals plus round-trip translations through Czech, Spanish, German and French. The contexts were left untranslated.
  • Annotation design:
    • 25 HITs of 10 sentences each, with 20 annotators per HIT.
    • A 4-point naturalness scale, from bad (1) to good (4).
    • Disjoint worker groups for each of the three conditions (H∅, H+, H−).
    • Filtering removed 504 ratings, about 3.9% of the total.
  • Correlations between conditions (Pearson's r): ratings with real context (H+) correlated at 0.940 with no-context ratings (H∅), and ratings with random context (H−) correlated at 0.911 with H∅. The two context conditions, H− and H+, correlated at 0.891 with each other.
  • Compression effect: when context was added, highly rated sentences dropped by about 0.08 on average, while poorly rated ones rose by about 0.45.
  • Coherence effect: real context gave consistently higher ratings than random context (one-tailed Wilcoxon signed-rank test, p = 1.9 × 10⁻⁸). Bidirectional models outperformed unidirectional ones (e.g., TDLM) by a wide margin.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓Queen Mary University of LondonLondon · United Kingdom54 works · 3,542 citations🎓University of GothenburgGothenburg · Sweden34 works · 1,625 citations🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations🔬Jožef Stefan InstituteLjubljana · Slovenia17 works · 2,165 citations🎓Jožef Stefan International Postgraduate SchoolLjubljana · Slovenia11 works · 1,776 citations🎓University of Nottingham Ningbo ChinaNingbo · China3 works · 216 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

Cite

Cite as Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver and Chang Shu. 2020. How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00315

@article{lau2020how,
  title = {{How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context}},
  author = {Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver and Chang Shu},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00315},
  url = {https://doi.org/10.1162/tacl_a_00315}
}

References

50 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adger. 2003. Core Syntax: A Minimalist Approach, Oxford University Press, United Kingdom. find
  2. John R. Anderson. 1996. ACT: A simple theory of complex cognition. American Psychologist, 51:355–365. \#\# 6 Conclusions and Future Work We found that processing context induces a cognitive load for humans, which creates a compression effect on the distribution of acceptability ratings. We also showed… find
  3. Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation, Cambridge University Press. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155.
  5. Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Minneapolis, Minnesota. find
  6. Jean-Philippe Bernardy, Shalom Lappin, and Jey Han Lau. 2018. The influence of context on sentence acceptability judgements. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 456–461. Melbourne, Australia. find
  7. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  8. Thomas G. Bever. 1970, The cognitive basis for linguistic structures, J. R. Hayes, editor, Cognition and the Development of Language, Wiley, New York, pages 279–362. find
  9. Maria Giavazzi, Sara Sambin, Ruth de Diego-Balaguer, Lorna Le Stanc, Anne-Catherine Bachoud-Lévi, and Charlotte Jacquemot. 2018. Structural priming in sentence comprehension: A single prime is enough. PLoS ONE, 13(4):e0194959. find
  10. Yuri Bizzoni and Shalom Lappin. 2019. The effect of context on metaphor paraphrase aptness judgments. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 165–175. Gothenburg, Sweden. find
  11. Dwight Bolinger. 1968. Judgments of grammaticality. Lingua, 21:34–40. find
  12. Jonathan Ginzburg. 2012. The Interactive Stance: Meaning for Conversation, Oxford University Press. find
  13. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. CoRR, abs/1906.09833. find
  14. Sidney Greenbaum. 1976. Contextual influence on acceptability judgements. Linguistics, 15(187):5–12. find
  15. Ronnie Cann, Ruth Kempson, and Matthew Purver. 2007. Context and well-formedness: the dynamics of ellipsis. Research on Language and Computation, 5(3):333–358. find
  16. Jeroen Groenendijk and Martin Stokhof. 1990. Dynamic Montague grammar. L. Kalman and L. Polos, editors, In Proceedings of the 2nd Symposium on Logic and Language, pages 3–48. Budapest. find
  17. Mickaël Causse, Vsevolod Peysakhovich, and Eve F. Fabre. 2016. High working memory load impairs language processing during a simulated piloting task: An ERP and pupillometry study. Frontiers in Human Neuroscience, 10:240. find
  18. Irene Heim. 1982. The Semantics of Definite and Indefinite Noun Phrases. Ph.D. thesis, University of Massachusetts at Amherst. find
  19. Wayne Cowart. 1994. Anchoring and grammar effects in judgments of sentence acceptability. Perceptual and Motor Skills, 79(3):1171–1182. find
  20. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860. find
  21. David Denison. 2004. Fuzzy Grammar: A Reader, Oxford University Press, United Kingdom. find
  22. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41:665–695.
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735–1780. find
  24. Aine Ito, Martin Corley, and Martin J. Pickering. 2018. A cognitive load delays predictive eye movements similarly during L1 and L2 comprehension. Bilingualism: Language and Cognition, 21(2):251–264. find
  25. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hans Kamp and Uwe Reyle. 199…
  26. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294. Association for Computational Linguisti… find
  27. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
  28. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, pages 1045–1048. Makuhari, Japan. find
  29. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
  30. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. Brussels, Belgium.
  31. Hyangsook Park, Jun-Su Kang, Sungmook Choi, and Minho Lee. 2013. Analysis of cognitive load for language processing based on brain activities. In Neural Information Processing, pages 561–568. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  32. Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959–968. Jeju Island, Korea. find
  33. Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017a. Topically driven neural language model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 355–365. Vancouver, Canada. find
  34. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  35. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2015. Unsupervised prediction of acceptability judgements. In Proceedings of the Joint conference of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of… find
  36. Daivd Reitter, Frank Keller, and Johanna D. Moore. 2011. A computational cognitive model of syntactic priming. Cognitive Science, 35(4):587–637. find
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Berlin, Germany.
  38. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017b. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive Science, 41:1202–1241. find
  39. Antonella Sorace and Frank Keller. 2005. Gradience in linguistic data. Lingua, 115:1497–1524. find
  40. William Mann and Sandra Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281.
  41. Jon Sprouse. 2007. Continuous acceptability, categorical grammaticality, and experimental syntax. Biolinguistics, 1123–134. find
  42. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Asso-John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2):257–285.
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008.
  44. Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural… find
  45. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36. Association for Computational Linguistics, Minneapolis, Minnesota.
  46. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
  47. John Warner and Arnold L. Glass. 1987. Context and distance-to-disambiguation effects in ambiguity resolution: Evidence from grammaticality judgments of garden path sentences. Journal of Memory and Language, 26(6):714 – 738. find
  48. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV…
  49. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural network acceptability judgments. CoRR, abs/1805.12471. find
  50. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Christina Zlogar and Kathryn Davidson. 2018. Effects of linguistic context on the acceptability of co-speech gestures. Glossa, 3(1):73. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lau-2020-how-furiously-can-colorless-green/