Elamur Research⌕
☰
home / papers / 2024 / Language Models as Inductive Reasoners

Language Models as Inductive Reasoners

Paper · 2024 · Zonglin Yang, Li Dong, Xinya Du et al.

Summary

Classical computational induction, such as Inductive Logic Programming, works over symbolic first-order logic. This makes it dependent on hand-preprocessed input, fragile when labels are wrong, and unable to treat synonymous expressions as equivalent. The authors reframe induction as a fully natural-language task: given facts written as ordinary text, a pretrained language model must generate a general rule that explains them.

To support this task they built DEER, a set of 1.2k rule–fact pairs. Each rule was written by an expert, and its supporting facts were gathered from web text. A companion resource, DEERLET, contains rules produced by GPT-J and annotated with labels for several quality aspects. The authors also propose automatic metrics that agree well with human judgments.

They then analyse how well PLMs induce rules across rule types, topics, numbers of input facts and model sizes. Finally, they introduce CoLM (chain-of-language-models), a framework grounded in philosophical criteria for valid induction, which beats baselines in both automatic and human evaluation. The work connects ILP and NLP and points toward LLM-assisted scientific hypothesis generation.

Key points

  • DEER pairs each rule with six supporting facts, three long paragraphs and three short core sentences, taken from search-engine results and Wikipedia.
  • DEER spans six topics: zoology, botany, geology, astronomy, history and physics. It also covers four FOL rule types: universal, existential, conjunctive and disjunctive implications.
  • In a manual check by four graduate students, 95.5% of facts contained enough information to induce their rule.
  • DEERLET stores (fact, rule, label0–label3) tuples, with rules produced by GPT-J through in-context learning. It is used to evaluate classifiers of specific rule qualities.
  • CoLM chains five PLM-based modules. M1 proposes candidate rules, and M2–M5 each test whether a rule satisfies one requirement of induction. Data and code are released at github.com/ZonglinY/Inductive_Reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States47 works · 5,363 citations🎓Nanyang Technological UniversitySingapore · Singapore73 works · 24,390 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao and Furu Wei. 2024. Language Models as Inductive Reasoners. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.eacl-long.13

@inproceedings{yang2024language,
  title = {{Language Models as Inductive Reasoners}},
  author = {Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao and Furu Wei},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.13},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.13}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2020. Identifying annotator bias: A new irt-based method for bias identification. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4787–4797. Internatio… find
  2. Aristotle Aristotle. 1994. Posterior analytics, volume 1. Clarendon Press Oxford, UK. find
  3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 2…
  4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.…
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…
  6. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3882–3890. ijcai.org.
  7. Andrew Cropper, Sebastijan Dumancic, Richard Evans, and Stephen H. Muggleton. 2022. Inductive logic programming at 30. Mach. Learn., 111(1):147–172. find
  8. Wang-Zhou Dai and Stephen H. Muggleton. 2021. Abductive knowledge induction from raw data. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 1845–1851. ijcai.org. find
  9. Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew McCallum. 2022. Knowledge base question answering by case-based reasoning over subgraphs. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland… find
  10. Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process… find
  11. Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. Commonsense knowledge mining from pre-trained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong…
  12. Luc De Raedt. 2010. Inductive logic programming. find
  13. Peter A Flach and Antonis C Kakas. 2000. Abductive and inductive reasoning: background and issues. In Abduction and induction, pages 1–27. Springer. find
  14. Evan Heit. 2000. Properties of inductive reasoning. Psychonomic Bulletin & Review, 7(4):569–592. find
  15. Karen Spärck Jones. 2004. A statistical interpretation of term specificity and its application in retrieval. J. Documentation, 60(5):493–502. find
  16. Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find
  17. Yiyang Li and Hai Zhao. 2023. EM pre-training for multi-party dialogue response generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 92–103. Association for Computational Ling… find
  18. Yufei Li, Zexin Li, Yingfan Gao, and Cong Liu. 2023. White-box multi-objective adversarial attack on dialogue generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1778–1792. A… find
  19. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael D. Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Nat…
  20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586.
  21. Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNL… find
  22. Hugo Mercier. 2018. The enigma of reason. In The enigma of reason. Harvard University Press. find
  23. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings…
  24. Kanishka Misra, Julia Taylor Rayz, and Allyson Ettinger. 2022. A property induction framework for neural language models. CoRR, abs/2205.06910. — [arXiv:2205.06910](https://arxiv.org/abs/2205.06910) find
  25. Stephen H. Muggleton and Luc De Raedt. 1994. Inductive logic programming: Theory and methods. J. Log. Program., 19/20:629–679. find
  26. Stephen H. Muggleton, Luc De Raedt, David Poole, Ivan Bratko, Peter A. Flach, Katsumi Inoue, and Ashwin Srinivasan. 2012. ILP turns 20 - biography and future challenges. Mach. Learn., 86(1):3–23. find
  27. John D Norton. 2005. A little survey of induction. find
  28. Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, Dublin, Ireland, December 15-18, 2020, pages 138–147. Association f… find
  29. Samir Okasha. 2002. Philosophy of science: A very short introduction, volume 67. Oxford Paperbacks. find
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
  31. Linlu Qiu and Liwei Jiang. 2023. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  33. Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th Edition). Pearson. find
  34. Merrilee H Salmon. 1989. Introduction to logic and critical thinking. find
  35. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… find
  36. Raymond M Smullyan. 1995. First-order logic. Courier Corporation. find
  37. Komal K. Teru, Etienne G. Denis, and William L. Hamilton. 2020. Inductive relation prediction by subgraph reasoning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9448–… find
  38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  39. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. find
  40. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023a. Learning to generate novel scientific directions with contextualized literature-based discovery. arXiv preprint arXiv:2305.14259. find
  41. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. 2023b. Hypothesis search: Inductive reasoning with language models. CoRR, abs/2309.05660. find
  42. Kaiyu Yang and Jia Deng. 2021. Learning symbolic rules for reasoning in quasi-natural language. CoRR, abs/2111.12038. — [arXiv:2111.12038](https://arxiv.org/abs/2111.12038) find
  43. Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong, Aline Villavicencio, and Xiaohui Cui. 2023. Evaluating open-domain dialogues in latent space with next sentence prediction and mutual information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  44. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven discovery of distributional differences via language descriptions. arXiv preprint arXiv:2302.14233. find
  45. Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. 2023. Large language models can learn rules. arXiv preprint arXiv:2310.07064. — [arXiv:2310.07064](https://arxiv.org/abs/2310.07064) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/