Elamur Research⌕
☰
home / papers / 2024 / ARN: Analogical Reasoning on Narratives

ARN: Analogical Reasoning on Narratives

Paper · 2024 · Zhivar Sourati, Filip Ilievski, Pia Sommerauer et al.

Summary

Most NLP research on analogy tests word-pair proportions (A : B :: C : D). Cognitive psychology, by contrast, studies analogy mainly through stories and separates shallow overlap from deeper relational structure. The authors ask whether current LLMs can detect such structure-level ("system") analogies between narratives.

They build a three-stage framework:

  • narrative elements are extracted using Gardner's (2003) narratology categories;
  • story pairs are connected by surface mappings (shared characters, settings, actions) or system mappings (a common high-level, proverb-like message);
  • the mix of mappings determines whether a pair is a near or far analogy or disanalogy.

This yields ARN, a binary task in which a model must choose the true analog of a query story over a distractor. In zero-shot tests, models handle near analogies well but fail on far ones, and GPT4.0 drops below chance there. Solved examples and Chain-of-Thought prompting help on far cases. Even so, the best few-shot score sits only midway between random and human performance, which reveals a clear gap in LLM analogical reasoning.

Key points

  • ARN contains 1,096 triples (query narrative, analogy, distractor). They are split into four balanced partitions: near/far analogies and near/far disanalogies. The benchmark is publicly released.
  • The evaluated models include Llama-2, GPT3.5 and GPT4.0, tested zero-shot and with 2, 4 or 6 demonstrations. GPT4.0 is best on average but scores below random on far analogies in the zero-shot setting.
  • Ordinary demonstrations improve far-analogy detection but hurt near-analogy accuracy. Demonstrations built from far analogies proved slightly more useful overall.
  • In the Chain-of-Thought prompts, the model first states each story's proverbial message and then answers. This raised far-analogy accuracy of GPT-based models by up to 30 absolute points over zero-shot.
  • Published in TACL vol. 12 (2024), pp. 1063–1086, by authors from USC's Information Sciences Institute and Vrije Universiteit Amsterdam.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States175 works · 37,047 citations🎓Vrije Universiteit AmsterdamAmsterdam · Netherlands43 works · 3,372 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Zhivar Sourati, Filip Ilievski, Pia Sommerauer and Yifan Jiang. 2024. ARN: Analogical Reasoning on Narratives. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00688

@article{sourati2024arn,
  title = {{ARN: Analogical Reasoning on Narratives}},
  author = {Zhivar Sourati, Filip Ilievski, Pia Sommerauer and Yifan Jiang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00688},
  url = {https://doi.org/10.1162/tacl_a_00688}
}

References

71 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandra Alexieva and Penka Hristova. 2017. Processing differences between near and far analogies. find
  2. Berkeley R. Andrus, Yeganeh Nasiri, Shilong Cui, Benjamin Cullen, and Nancy Fulda. 2022. Enhanced story comprehension for large language models through dynamic document-based knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10436–10444. https://do… — [doi:10.1609/aaai.v36i10.21286](https://doi.org/10.1609/aaai.v36i10.21286) find
  3. Mieke Bal. 2017. Narratology: Introduction to the Theory of Narrative, 4th edition. University of Toronto Press. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… — Language Models are Few-Shot Learners
  5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  6. Nigel H. Collier, Fangyu Liu, and Ehsan Shareghi. 2023. On reality and the limits of language data: Aligning llms with human norms. find
  7. Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. Scientific and creative analogies in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094–2100, Abu Dhabi, United Arab Emirates. Association for Computation… find
  8. Kevin N. Dunbar and David Klahr. 2012. 701 Scientific Thinking and Reasoning. In The Oxford Handbook of Thinking and Reasoning. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199734689 .013.0035 — [doi:10.1093/oxfordhb/9780199734689](https://doi.org/10.1093/oxfordhb/9780199734689) ↗
  9. Michael C. Frank. 2023. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452. https://doi.org/10.1038/s44159-023 -00211-x Bin Gao, Jiang Bian, and Tie-Yan Liu. 2014. Wordrep: A benchmark for research on learning word representations. arXiv preprin… — [doi:10.1038/s44159-023](https://doi.org/10.1038/s44159-023) ↗
  10. Janet Gardner. 2003. Writing about Literature. Bedford Books. find
  11. D. Gentner, L. Smith, and V. S. Ramachandran. 2012. Analogical reasoning, 2012. Encyclopedia of Human Behavior, 2nd ed., VS Ramachandran, ed., Elsevier, Oxford, UK, pages 130–136. https://doi.org/10.1016/B978-0-12-375000-6.00022-7 ↗
  12. Dedre Gentner. 1982. Are scientific analogies metaphors. Metaphor: Problems and Perspectives, 7(7). find
  13. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155–170. https://doi.org /10.1016/S0364-0213(83)80009-3 ↗
  14. Dedre Gentner and Mary Jo Rattermann. 1991. Language and the Career of Similarity. Cambridge University Press. https://doi.org/10.1017/CBO9780511983689.008 — [doi:10.1017/cbo9780511983689.008](https://doi.org/10.1017/cbo9780511983689.008) ↗
  15. Dedre Gentner, Mary Jo Rattermann, and Kenneth D. Forbus. 1993. The roles of similarity in transfer: Separating retrievability from inferential soundness. Cognitive Psychology, 25(4):524–575. https://doi.org/10.1006/cogp.1993.1013, PubMed: 8243045 ↗
  16. Dedre Gentner and Cecile Toupin. 1986. Systematicity and surface similarity in the development of analogy. Cognitive Science, 10(3):277–300. https://doi.org/10.1016 /S0364-0213(86)80019-2 find
  17. Sayan Ghosh and Shashank Srivastava. 2022. ePiC: Employing proverbs in context as a benchmark for abstract language understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3989–4004, Dublin, Ireland. Association for Com… find
  18. Mary L. Gick and Keith J. Holyoak. 1980. Analogical problem solving. Cognitive Psychology, 12(3):306–355. https://doi.org/10.1016 /0010-0285(80)90013-4 find
  19. Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn’t. In Proceedings of the NAACL Student Research Workshop, pages 8–15. https://doi.org/10.18653/v1/N16-2002 ↗
  20. Adam E. Green, Jonathan A. Fugelsang, and Kevin N. Dunbar. 2006. Automatic activation of categorical and abstract analogical relations in analogical reasoning. Memory & Cognition, 34:1414–1421. https://doi.org/10.3758/BF03195906, PubMed: 17263066 ↗
  21. Adam E. Green, Jonathan A. Fugelsang, David J. M. Kraemer, and Kevin N. Dunbar. 2008. The micro-category account of analogy. Cognition, 106(2):1004–1016. https://doi.org/10 .1016/j.cognition.2007.03.015, PubMed: 17511980 find
  22. Adam E. Green, David J. M. Kraemer, Jonathan A. Fugelsang, Jeremy R. Gray, and Kevin N. Dunbar. 2010. Connecting long distance: Semantic distance in analogical reasoning modulates frontopolar cortex activity. Cerebral Cortex, 20(1):70–76. https://doi.org/10 .1093/cercor/bhp081, PubMed: 19383937 find
  23. Graeme S. Halford. 1992. Analogical reasoning and conceptual complexity in cognitive development. Human Development, 35(4):193–217. https://doi.org/10.1159/000277167 — [doi:10.1159/000277167](https://doi.org/10.1159/000277167) ↗
  24. Mary Hesse. 1965. Models and analogies in science. find
  25. Douglas R. Hofstadter. 2001. Analogy as the core of cognition. The Analogical Mind: Perspectives from Cognitive Science, pages 499–538. https://doi.org/10.7551/mitpress/1251.003.0020 — [doi:10.7551/mitpress](https://doi.org/10.7551/mitpress) ↗
  26. Keith J. Holyoak. 2012. 234 Analogy and relational reasoning. In The Oxford Handbook of Thinking and Reasoning. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199734689.013.0013 ↗
  27. K. J. Holyoak and P. Thagard. 1996. Mental Leaps: Analogy in Creative Thought. A Bradford book. MIT Press. find
  28. David Patrick Houghton. 1998. Analogical reasoning and policymaking: Where and when is it used? Policy Sciences, 31(3):151–176. https://doi.org/10.1023/A:1004355215177 — [doi:10.1023/a:1004355215177](https://doi.org/10.1023/a:1004355215177) ↗
  29. Nicholas Ichien, Hongjing Lu, and Keith J. Holyoak. 2020. Verbal analogy problem sets: An inventory of testing materials. Behavior Research Methods, 52(5):1803–1816. https://doi.org/10.3758/s13428-019-01312-3, PubMed: 31898298 — [doi:10.3758/s13428-019-01312-3](https://doi.org/10.3758/s13428-019-01312-3) ↗
  30. Molly E. Ireland and James W. Pennebaker. 2010. Language style matching in writing: Synchrony in essays, correspondence, and poetry. Journal of Personality and Social Psychology, 99(3):549. https://doi.org/10.1037/a0020386, PubMed: 20804263 ↗
  31. Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, and Zheng Zhang. 2023. StoryAnalogy: Deriving story-level analogies from large language models to unlock analogical understanding. In Proceedings of the 2023 Con… find
  32. David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012. SemEval-2012 task 2: Measuring degrees of relational similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proc…
  33. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNI-FIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Co… — [doi:10.18653/v1/2020.findings](https://doi.org/10.18653/v1/2020.findings) find
  34. Vince Kotchian and Curt Simmons. 2012. SSAT and ISEE For Dummies. John Wiley & Sons, Inc. find
  35. Martha Lewis and Melanie Mitchell. 2024. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv preprint arXiv:2402.08955v1. — [arXiv:2402.08955](https://arxiv.org/abs/2402.08955) find
  36. Hongjing Lu, Nicholas Ichien, and Keith J. Holyoak. 2022. Probabilistic analogical mapping with semantic relation networks. Psychological Review. https://doi.org/10 .1037/rev0000358, PubMed: 35389714 find
  37. I. Mani. 2013. Computational Modeling of Narrative. Synthesis Lectures on Human Language Technologies. Morgan & Claypool. https://doi.org/10.1007/978-3-031 -02147-3 ↗
  38. Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276–282. https://doi.org/10.11613/BM.2012.031, PubMed: 23092060 ↗
  39. Wolfgang Mieder. 1993. Proverbs Are Never Out of Season: Popular Wisdom in the Modern Age. Oxford University Press New York, New York. find
  40. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781v3.
  41. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26.
  42. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia.…
  43. Melanie Mitchell. 2021. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79–101. https://doi.org/10.1111/nyas.14619, PubMed: 34173249 ↗
  44. Melanie Mitchell, Alessandro B. Palmarini, and Arseny Moskvichev. 2023. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247v3. — [arXiv:2311.09247](https://arxiv.org/abs/2311.09247) find
  45. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  46. Thiloshon Nagarajah, Filip Ilievski, and Jay Pujara. 2022. Understanding narratives through dimensions of analogy. find
  47. OpenAI. 2022. Chatgpt. https://openai.com/blog/chatgpt. Accessed: April 30, 2023. ↗
  48. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  49. Derek C. Penn, Keith J. Holyoak, and Daniel J. Povinelli. 2008. Darwin’s mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and Brain Sciences, 31(2):109–130. https://doi.org/10.1017/S0140525X08003543, PubMed: 18479531 ↗
  50. David Premack. 1983. The codes of man and beasts. Behavioral and Brain Sciences, 6(1):125–136. https://doi.org/10.1017/S0140525X00015077 ↗
  51. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. volume 1, page 9. find
  52. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  53. Claire E. Stevenson, Mathilde ter Veen, Rochelle Choenni, Han L. J. van der Maas, and Ekaterina Shutova. 2023. Do large language models solve verbal analogies like children do? arXiv preprint arXiv:2310.20384v1. find
  54. Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Chai. 2021. Tiered reasoning for intuitive physics: Toward verifiable commonsense language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4902–4918, Punta Cana, Dominican Republic. Association for Comput… find
  55. Oren Sultan and Dafna Shahaf. 2022. Life is a circus and we are the clowns: Automatically finding analogies between situations and processes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3547–3562, Abu Dhabi, United Arab Emirates. Association for C… find
  56. Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with macaw. arXiv preprint arXiv:2109.02593v1. find
  57. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  58. Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules to solve multiple-choice synonym and analogy problems. arXiv preprint cs/0309035v1. find
  59. Varsity Tutors. Analogies - ssat elementary level verbal. find
  60. Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2021. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint C… find
  61. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, 30.
  62. Michael S. Vendetti, Aaron Wu, and Keith J. Holyoak. 2014. Far-out thinking: Generating solutions to distant analogies promotes relational thinking. Psychological Science, 25(4):928–933. https://doi.org/10.1177/0956797613518079, PubMed: 24463552 ↗
  63. Piek Vossen, Tommaso Caselli, and Roxane Segers. 2021. A narratology-based framework for storyline extraction. Computational Analysis of Storylines: Making Sense of Events, 125. https://doi.org/10.1017/9781108854221.008 ↗
  64. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541. https://doi.org/10.1038/s41562-023-01659-w, PubMed: 37524930 ↗
  65. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. Transactions on M…
  66. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  67. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-shot information extraction via chatting with chatgpt. find
  68. C. M. Wharton, K. J. Holyoak, P. E. Downing, T. E. Lange, T. D. Wickens, and E. R. Melz. 1994. Below the surface: Analogical similarity and retrieval competition in reminding. Cognitive Psychology, 26(1):64–101. https://doi.org/10.1006/cogp.1994.1003 — [doi:10.1006/cogp.1994.1003](https://doi.org/10.1006/cogp.1994.1003) ↗
  69. Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, Aman Chadha, Aishwarya Naresh Reganti, Amit Sheth, and Amitava Das. 2023. ANALOGICAL - A novel benchmark for long text analogy evaluation in large language models. In Findings of the Association for Comp… find
  70. A Data Collection Details As we use the ePiC dataset (Ghosh and Srivastava, 2022) as our narrative database, there are certain pre-processings that needed to be done on the original ePiC dataset before they would be ready for the subsequent steps explained in Section 4. find
  71. Proverb Practice what you preach Chain is only as strong as its weakest link A miss is as good as a mile find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sourati-2024-arn-analogical-reasoning-narratives/