Elamur Research⌕
☰
home / papers / 2022 / Language models show human-like content effects on reasoning tasks

Language models show human-like content effects on reasoning tasks

Paper · 2022 · Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan et al.

Summary

Ideally, abstract reasoning should not depend on what a problem is about. Yet people reason better when a problem's meaning agrees with the logically correct answer. Psychologists call these "content effects." This paper asks whether large language models, whose expectations come from human-written text, blend meaning and logic in the same way.

To avoid training-data contamination, the authors built fresh stimuli for three tasks:

  • a forced-choice Natural Language Inference task about comparisons
  • judging whether syllogisms are valid
  • the Wason selection task

Each task has belief-consistent or realistic versions, belief-violating or arbitrary versions, and versions using nonsense words. Several LMs were scored by answer likelihood. Online human participants solved the same problems with matching instructions.

Both groups were more accurate when content supported the right inference. The effect was modest on NLI and stronger on syllogisms and Wason. Model answer distributions also tracked human response times. The results suggest such biases can emerge from learning on naturalistic data, which matters for cognitive theory and for interpreting LM performance.

Key points

  • Models tested were Chinchilla (70B), PaLM 2-M and PaLM 2-L, Flan-PaLM 2, and GPT-3.5-turbo-instruct. Content effects were similar across base and instruction-tuned models.
  • Syllogism and Wason answers were scored with the DC-PMI correction (Holtzman et al.). NLI used plain maximum likelihood, because DC-PMI would cancel the bias being measured. The direction of the effects mostly held under alternative scoring.
  • The human data were collected in 2023 from native English speakers in the UK. The first sample had 625 people. A second sample of 360 people focused on Wason only and offered a £0.50 accuracy bonus because Wason accuracy was low.
  • Nonsense-word controls (e.g., "vuffs," "feps") remove semantic content entirely. The analyses used mixed-effects logistic regression with random effects for items and models.
  • For library citation: Andrew K. Lampinen and Ishita Dasgupta are equal-contribution first authors (Google DeepMind). Lampinen is listed first in arXiv:2207.07051.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. arXiv:2207.07051.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2207.07051 · arXiv 2207.07051

@misc{dasgupta2022language,
  title = {{Language models show human-like content effects on reasoning tasks}},
  author = {Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland and Felix Hill},
  year = {2022},
  doi = {10.48550/arxiv.2207.07051},
  eprint = {2207.07051},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.07051}
}

References

120 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. C. Wason, Reasoning about a rule. Quarterly Journal of Experimental Psychology 20, 273 - 281 (1968). find
  2. A. Newell, Physical symbol systems. Cognitive science 4, 135–183 (1980). find
  3. J. A. Fodor, Z. W. Pylyshyn, Connectionism and cognitive architecture: A critical analysis. Cognition 28, 3–71 (1988). find
  4. G. Marcus, The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177 (2020).
  5. M. Mitchell, Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences 1505, 79–101 (2021).
  6. J. Russin, R. C. O’Reilly, Y. Bengio, Deep learning needs a prefrontal cortex. Work Bridging AI Cogn Sci 107, 603–616 (2020). find
  7. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020).
  8. D. Ganguli, D. Hernandez, L. Lovitt, N. DasSarma, T. Henighan, A. Jones, N. Joseph, J. Kernion, B. Mann, A. Askell, et al., Predictability and surprise in large generative models. arXiv preprint arXiv:2202.07785 (2022).
  9. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al., Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 (2021).
  10. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022).
  11. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).
  12. R. Schaeffer, B. Miranda, S. Koyejo, Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004 (2023).
  13. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al., Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021). find
  14. Y. Razeghi, R. L. Logan IV, M. Gardner, S. Singh, Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206 (2022). — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) find
  15. A. Patel, S. Bhattamishra, N. Goyal, Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021).
  16. K. Valmeekam, A. Olmo, S. Sreedharan, S. Kambhampati, Large language models still can’t plan (a benchmark for llms on planning and reasoning about change) (2022). find
  17. G. Gigerenzer, W. Gaissmaier, Heuristic decision making. Annual review of psychology 62, 451–482 (2011). find
  18. D. Kahneman, S. P. Slovic, P. Slovic, A. Tversky, Judgment under uncertainty: Heuristics and biases (Cambridge university press, 1982). find
  19. G. Marcus, Kluge: The haphazard evolution of the human mind (Houghton Mifflin Harcourt, 2009). find
  20. J. Evans, J. L. Barston, P. Pollard, On the conflict between logic and belief in syllogistic reasoning. Memory & cognition 11, 295–306 (1983). find
  21. J. S. B. Evans, T. S. Perry, Belief bias in children’s reasoning. Cahiers de Psychologie Cognitive/Current Psychology of Cognition (1995). find
  22. K. C. Klauer, J. Musch, B. Naumer, On belief bias in syllogistic reasoning. Psychological review 107, 852 (2000). find
  23. P. N. Johnson-Laird, Deductive reasoning. Annual review of psychology 50, 109–135 (1999). find
  24. P. W. Cheng, K. J. Holyoak, Pragmatic reasoning schemas. Cognitive psychology 17, 391–416 (1985). find
  25. L. Cosmides, The logic of social exchange: Has natural selection shaped how humans reason? studies with the wason selection task. Cognition 31, 187–276 (1989). find
  26. L. Cosmides, J. Tooby, Cognitive adaptations for social exchange. The adapted mind: Evolutionary psychology and the generation of culture 163, 163–228 (1992). find
  27. A. L. Cohen, S. Sidlowski, A. Staub, Beliefs and bayesian reasoning. Psychonomic Bulletin & Review 24, 972–978 (2017). find
  28. I. Dasgupta, E. Schulz, J. B. Tenenbaum, S. J. Gershman, A theory of learning to infer. Psychological review 127, 412 (2020). find
  29. D. J. Benjamin, Errors in probabilistic reasoning and judgment biases, Tech. rep., National Bureau of Economic Research (2018). find
  30. C. R. Peterson, Z. Ulehla, Uncertainty, inference difficulty, and probability learning. Journal of Experimental Psychology 67, 523–530 (1964). find
  31. P. N. Johnson-Laird, P. Legrenzi, M. S. Legrenzi, Reasoning and a sense of reality. British journal of Psychology 63, 395–400 (1972). find
  32. P. C. Wason, P. N. Johnson-Laird, Psychology of reasoning: Structure and content, vol. 86 (Harvard University Press, 1972). find
  33. J. S. B. Evans, Bias in human reasoning: Causes and consequences. (Lawrence Erlbaum Associates, Inc, 1989). find
  34. G. F. Marcus, The algebraic mind: Integrating connectionism and cognitive science (MIT press, 2003). find
  35. J. L. McClelland, M. M. Botvinick, D. C. Noelle, D. C. Plaut, T. T. Rogers, M. S. Seidenberg, L. B. Smith, Letting structure emerge: connectionist and dynamical systems approaches to cognition. Trends in cognitive sciences 14, 348–356 (2010). find
  36. A. Santoro, A. Lampinen, K. Mathewson, T. Lillicrap, D. Raposo, Symbolic behaviour in artificial intelligence. arXiv preprint arXiv:2102.03406 (2021). — [arXiv:2102.03406](https://arxiv.org/abs/2102.03406) find
  37. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).
  38. S. Bhatia, R. Richie, W. Zou, Distributed semantic representations for modeling human judgment. Current Opinion in Behavioral Sciences 29, 31–36 (2019). find
  39. S. Bhatia, R. Richie, Transformer networks of human conceptual knowledge. Psychological Review (2021). find
  40. T. H. Trinh, Q. V. Le, A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847 (2018). find
  41. F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, A. Miller, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019), pp. 2463–2473. find
  42. L. Z. Liu, Y. Wang, J. Kasai, H. Hajishirzi, N. A. Smith, Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.07885 (2021).
  43. Z. Jiang, J. Araki, H. Ding, G. Neubig, How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9, 962–977 (2021). find
  44. A. Søgaard, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021), pp. 8240–8245. find
  45. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022).
  46. Gpt-3.5, https://platform.openai.com/docs/models/gpt-3-5. trieved September 19th, 2023. ↗
  47. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al., Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023).
  48. B. MacCartney, C. D. Manning, Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (2007), pp. 193–200. find
  49. S. R. Bowman, G. Angeli, C. Potts, C. D. Manning, A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 (2015).
  50. I. Dasgupta, D. Guo, A. Stuhlmüller, S. J. Gershman, N. D. Goodman, Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302 (2018). — [arXiv:1802.04302](https://arxiv.org/abs/1802.04302) find
  51. R. Smith, The Stanford Encyclopedia of Philosophy, E. N. Zalta, ed. (Metaphysics Research Lab, Stanford University, 2020), Fall 2020 edn. find
  52. M. H. Tessler, J. B. Tenenbaum, N. D. Goodman, Logic, probability, and pragmatics in syllogistic reasoning. Topics in Cognitive Science (2022). Re- find
  53. C. Dube, C. M. Rotello, E. Heit, Assessing the belief bias effect with rocs: it’s a response bias effect. Psychological review 117, 831 (2010). find
  54. D. Trippas, M. F. Verde, S. J. Handley, Using forced choice to test belief bias in syllogistic reasoning. Cognition 133, 586–600 (2014). find
  55. M. H. Tessler, Understanding belief bias by measuring prior beliefs for a bayesian model of syllogistic reasoning. Proceedings of ESSLLI pp. 225–237 (2015). find
  56. P. C. Wason, D. Shapiro, Natural and contrived experience in a reasoning problem. Quarterly journal of experimental psychology 23, 63–71 (1971). find
  57. M. Inglis, A. Simpson, Mathematicians and the selection task. International Group for the Psychology of Mathematics Education (2004). find
  58. C. Cresswell, C. P. Speelman, Does mathematics training lead to better logical thinking and reasoning? a cross-sectional assessment from students to professors. PLOS ONE 15, 1-21 (2020). find
  59. J. S. B. Evans, J. Curtis-Holmes, Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning. Thinking & Reasoning 11, 382–389 (2005). find
  60. J. S. B. Evans, S. Newstead, J. Allen, P. Pollard, Debiasing by instruction: The case of belief bias. European Journal of Cognitive Psychology 6, 263–285 (1994). find
  61. W. A. Wickelgren, Speed-accuracy tradeoff and information processing dynamics. Acta psychologica 41, 67–85 (1977). find
  62. S. L. Wise, X. Kong, Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education 18, 163–183 (2005). find
  63. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. H. Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndouss… find
  64. A. Goodkind, K. Bicknell, Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018) (2018), pp. 10–18. find
  65. R. Futrell, E. Wilcox, T. Morita, P. Qian, M. Ballesteros, R. Levy, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Association for Computational Linguistics, Minneapolis… find
  66. M. Binz, E. Schulz, Using cognitive psychology to understand gpt-3 (2022). find
  67. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, D. Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022).
  68. D. Khashabi, C. Baral, Y. Choi, H. Hajishirzi, Findings of the Association for Computational Linguistics: ACL 2022 (2022), pp. 589–612. find
  69. A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, F. Hill, Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 (2022).
  70. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, D. Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022).
  71. H. Mercier, D. Sperber, The Enigma of Reason (Harvard University Press, 2017). find
  72. J. S. B. Evans, Heuristic and analytic processes in reasoning. British Journal of Psychology 75, 451–468 (1984). find
  73. J. S. B. Evans, In two minds: dual-process accounts of reasoning. Trends in cognitive sciences 7, 454–459 (2003). find
  74. M. Oaksford, N. Chater, Optimal data selection: Revision, review, and reevaluation. Psychonomic Bulletin & Review 10, 289–318 (2003). find
  75. D. Kahneman, Thinking, fast and slow (Macmillan, 2011). find
  76. J. S. B. Evans, D. E. Over, Rationality and reasoning (Psychology Press, 2013). find
  77. Y. Bengio, The consciousness prior. arXiv preprint arXiv:1709.08568 (2017). — [arXiv:1709.08568](https://arxiv.org/abs/1709.08568) find
  78. M. Nye, M. Tessler, J. Tenenbaum, B. M. Lake, Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems 34, 25192–25204 (2021). find
  79. J. D. Cohen, K. Dunbar, J. L. McClelland, On the control of automatic processes: a parallel distributed processing account of the stroop effect. Psychological review 97, 332 (1990). find
  80. M. M. Botvinick, J. D. Cohen, The computational and neural basis of cognitive control: charted territory and new frontiers. Cognitive science 38, 1249–1285 (2014). find
  81. J. Duncan, M. Assem, S. Shashidhara, Integrated intelligence from distributed brain activity. Trends in Cognitive Sciences 24, 838–852 (2020). find
  82. Y. Li, J. L. McClelland, A weighted constraint satisfaction approach to human goal-directed decision making. PLOS Computational Biology 18, e1009553 (2022). find
  83. D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, J. J. DiCarlo, Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences 111, 8619–8624 (2014). find
  84. D. L. Yamins, J. J. DiCarlo, Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience 19, 356–365 (2016). find
  85. R. Cao, D. Yamins, Explanatory models in neuroscience: Part 1–taking mechanistic abstraction seriously. arXiv preprint arXiv:2104.01490 (2021). — [arXiv:2104.01490](https://arxiv.org/abs/2104.01490) find
  86. R. Cao, D. Yamins, Explanatory models in neuroscience: Part 2–constraint-based intelligibility. arXiv preprint arXiv:2104.01489 (2021). — [arXiv:2104.01489](https://arxiv.org/abs/2104.01489) find
  87. M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, E. Fedorenko, The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences 118 (2021). find
  88. S. Kumar, T. R. Sumers, T. Yamakoshi, A. Goldstein, U. Hasson, K. A. Norman, T. L. Griffiths, R. D. Hawkins, S. A. Nastase, Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. bioRxiv (2022). find
  89. A. Goldstein, Z. Zada, E. Buchnik, M. Schain, A. Price, B. Aubrey, S. A. Nastase, A. Feder, D. Emanuel, A. Cohen, et al., Shared computational principles for language processing in humans and deep language models. Nature neuroscience 25, 369–380 (2022). find
  90. E. G. Wilcox, J. Gauthier, J. Hu, P. Qian, R. Levy, On the predictive power of neural language models for human real-time comprehension behavior. arXiv:2006.01912 (2020).
  91. F. Lieder, T. L. Griffiths, Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences 43 (2020). find
  92. S. J. Gershman, E. J. Horvitz, J. B. Tenenbaum, Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science 349, 273–278 (2015). find
  93. H. A. Simon, Utility and probability (Springer, 1990), pp. 15–18. find
  94. A. Webson, E. Pavlick, Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247 (2021).
  95. S. Mishra, D. Khashabi, C. Baral, H. Hajishirzi, Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773 (2021). find
  96. J. L. McClelland, F. Hill, M. Rudolph, J. Baldridge, H. Schütze, Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models. Proceedings of the National Academy of Sciences 117, 25966–25974 (2020). find
  97. M. W. Alibali, R. Boncoddo, A. B. Hostetter, Gesture in reasoning: An embodied perspective. The Routledge handbook of embodied cognition p. 150 (2014). find
  98. M. J. Nathan, K. E. Schenck, R. Vinsonhaler, J. E. Michaelis, M. I. Swart, C. Walkington, Embodied geometric reasoning: Dynamic gestures during intuition, insight, and proof. Journal of Educational Psychology (2020). arXiv preprint find
  99. H. H. Clark, Using language (Cambridge university press, 1996). find
  100. D. Schlangen, Norm participation grounds language. arXiv preprint arXiv:2206.02885 (2022). — [arXiv:2206.02885](https://arxiv.org/abs/2206.02885) find
  101. A. K. Luria, Towards the problem of the historical nature of psychological processes. International Journal of Psychology 6, 259–272 (1971). find
  102. D. R. Lehman, R. E. Nisbett, A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology 26, 952 (1990). find
  103. N. Attridge, A. Aberdein, M. Inglis, Does studying logic improve logical reasoning? (2016). find
  104. A. J. Nam, J. L. McClelland, What underlies rapid learning and systematic generalization in humans. arXiv preprint arXiv:2107.06994 (2021). — [arXiv:2107.06994](https://arxiv.org/abs/2107.06994) find
  105. P. Clark, O. Tafjord, K. Richardson, Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05867 (2020).
  106. Y. Wu, M. N. Rabe, W. Li, J. Ba, R. B. Grosse, C. Szegedy, International Conference on Machine Learning (PMLR, 2021), pp. 11251–11262. find
  107. T. Schick, S. Udupa, H. Schütze, Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics 9, 1408-1424 (2021).
  108. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, J. Schulman, Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021).
  109. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, J. Leike, Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 (2022).
  110. E. Zelikman, Y. Wu, N. D. Goodman, Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465 (2022).
  111. J. X. Wang, Meta-learning in natural and artificial intelligence. Current Opinion in Behavioral Sciences 38, 90–95 (2021). find
  112. S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClelland, F. Hill, Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems 35, 18878–18891 (2022). find
  113. B. Prystawski, N. D. Goodman, Why think step-by-step? reasoning emerges from the locality of experience. arXiv preprint arXiv:2304.03843 (2023). find
  114. A. Emami, A. Trischler, K. Suleman, J. C. K. Cheung, An analysis of dataset overlap on winograd-style tasks. arXiv preprint arXiv:2011.04767 (2020). find
  115. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, M. Gardner, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021), pp. 1286–1305. find
  116. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al., The Eleventh International Conference on Learning Representations (2022). find
  117. A. Holtzman, P. West, V. Shwartz, Y. Choi, L. Zettlemoyer, Surface form competition: Why the highest probability answer isn’t always right. arXiv preprint arXiv:2104.08315 (2021).
  118. A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi, International Conference on Learning Representations (2019). find
  119. A. Hosseini, S. Reddy, D. Bahdanau, R. D. Hjelm, A. Sordoni, A. Courville, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Association for Computational Linguistics, Online, 2021), pp. 1301–1312. find
  120. J. S. B. Evans, J. Clibbens, B. Rood, The role of implicit and explicit negation in conditional reasoning bias. Journal of Memory and Language 35, 392–409 (1996). Acknowledgements: We thank Michiel Bakker, Michael Henry Tessler, Matt Botvinick, and Adam Santoro for helpful comments and suggestions.… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dasgupta-2022-language-models-show-human-like/