Elamur Research⌕
☰
home / papers / 2023 / Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

Paper · 2023 · Risako Ando, Takanobu Morishita, Hirohiko Abe et al.

Summary

The paper asks whether large language models fall into the same reasoning traps that psychologists have documented in people solving syllogisms. The authors build NeuBAROCO, a bilingual English–Japanese resource derived from BAROCO, a question collection previously used to measure human deductive skill, including in twin studies. Each original multiple-choice item was recast in natural language inference format: two premises plus one candidate conclusion, hand-labelled as entailment, contradiction, or neutral. Items are also grouped as symbolic, belief-consistent, or belief-inconsistent, and problems where humans typically slip through conversion errors or atmosphere effects are marked. RoBERTa and BART (English) and GPT-3.5 (English and Japanese) were tested without any solved examples. No model proved dependable. GPT-3.5 answered only about half of the problems correctly. Every system scored lowest on belief-inconsistent items and on conversion and atmosphere cases, where RoBERTa and BART were close to zero. The work connects cognitive-science research on reasoning biases with LLM evaluation. It also shows that the bias pattern appears in two typologically distant languages.

Key points

  • NeuBAROCO has 375 inference problems: 122 entailment, 71 contradiction and 182 neutral. They were derived from BAROCO-ALL, a 200-question subset of BAROCO (Shikishima et al.).
  • The evaluation covers three human-like biases: belief bias, conversion errors and atmosphere effects. Problems are also split by content type (symbol / consistent / inconsistent).
  • Overall accuracy was:
    • RoBERTa: 34.67%
    • BART: 35.20%
    • GPT-3.5 (English): 51.73%
    • GPT-3.5 (Japanese): 48.27%
  • English GPT-3.5 handled entailment well (79.51%) but struggled with contradiction (38.03%) and neutral items (38.46%).
  • On bias-prone items, accuracy dropped sharply:
    • Conversion: 0% for RoBERTa, 1.43% for BART, 25.71% for GPT-3.5 (EN) and 21.43% for GPT-3.5 (JA).
    • Atmosphere: 0% for RoBERTa, 0.96% for BART, 39.42% for GPT-3.5 (EN) and 22.12% for GPT-3.5 (JA).
    • Belief-inconsistent: the weakest content category for every model, with GPT-3.5 (EN) at 31.37%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Keio UniversityTokyo · Japan7 works · 882 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima and Mitsuhiro Okada. 2023. Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases. arXiv:2306.12567.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.12567 · arXiv 2306.12567

@misc{ando2023evaluating,
  title = {{Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases}},
  author = {Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima and Mitsuhiro Okada},
  year = {2023},
  doi = {10.48550/arxiv.2306.12567},
  eprint = {2306.12567},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.12567}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Nick Chater and Mike Oaksford. 1999. The probability heuristics model of syllogistic reasoning. Cognitive Psychology, 38(2):191–258. find
  3. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  5. Jonathan St.B. T. Evans. 1989. Bias in Human Reasoning: Causes and Consequences. Lawrence Erlbaum Associates, Inc. find
  6. Jonathan St.B. T. Evans, Stephen E. Newstead, and Ruth M. J. Byrne. 1993. Human Reasoning: The Psychology of Deduction. Psychology Press. find
  7. Bart Geurts. 2003. Reasoning with quantifiers. Cognition, 86(3):223–251. find
  8. Vinod Goel, Christian Buchel, Chris Frith, and Raymond J Dolan. 2000. Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage, 12(5):504–514. find
  9. Sangeet Khemlani and Philip N Johnson-Laird. 2012. Theories of the syllogism: A meta-analysis. Psychological Bulletin, 138(3):427. find
  10. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  12. Lawrence S. Moss. 2015. Natural logic. In Shalom Lappin and Chris Fox, editors, The Handbook of Contemporary Semantic Theory, 2 edition, pages 559– 592. Wiley. find
  13. Stephen E. Newstead. 1989. Interpretational errors in syllogistic reasoning. Journal of Memory and Language, 28(1):78–91. find
  14. Stephen E Newstead, Paul Pollard, Jonathan St BT Evans, and Julie L Allen. 1992. The source of belief bias effects in syllogistic reasoning. Cognition, 45(3):257–284. find
  15. Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8713–8721.
  16. Víctor Sánchez Valencia. 1991. Studies on Natural Logic and Categorial Grammar. Ph.D. thesis, University of Amsterdam. find
  17. Yuri Sato and Koji Mineshima. 2015. How diagrams can support syllogistic reasoning: an experimental study. Journal of Logic, Language and Information, 24:409–455. find
  18. Viktor Schlegel, Kamen Pavlov, and Ian Pratt-Hartmann. 2022. Can transformers reason in fragments of natural language? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11184–11199, Abu Dhabi, United Arab Emirates. Association for Computational Linguist… find
  19. Chizuru Shikishima, Juko Ando, Pierre Grialou, Ryo Takemura, and Mitsuhiro Okada. 2006. A behavioral genetic study of syllogism solving using linguistic and graphical representations: a preliminary report. In Images and Reasoning, pages 69–85. Keio University Press. find
  20. Chizuru Shikishima, Kai Hiraishi, Shinji Yamagata, Juko Ando, and Mitsuhiro Okada. 2015. Genetic factors of individual differences in decision making in economic behavior: A Japanese twin study using the Allais problem. Frontiers in Psychology, 6:1712. find
  21. Chizuru Shikishima, Kai Hiraishi, Shinji Yamagata, Yutaro Sugimoto, Ryo Takemura, Koken Ozaki, Mitsuhiro Okada, Tatsushi Toda, and Juko Ando. 2009. Is g an entity? a Japanese twin study using syllogisms and intelligence tests. Intelligence, 37(3):256–267. find
  22. Keith Stenning and Michiel Van Lambalgen. 2012. Human Reasoning and Cognitive Science. MIT Press. find
  23. Johan van Benthem. 1986. Essays in Logical Semantics. Reidel, Dordrecht. find
  24. Minna Cheves Wilkins. 1928. The effect of changed material on ability to do formal syllogistic reasoning. Archives of Psychology, 16:1–83. find
  25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  26. Robert S Woodworth and Saul B. Sells. 1935. An atmosphere effect in formal syllogistic reasoning. Journal of Experimental Psychology, 18(4):451–460. find
  27. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/