Elamur Research⌕
☰
home / papers / 2026 / Comparing Human and Large Language Model Interpretation of Implicit Information

Comparing Human and Large Language Model Interpretation of Implicit Information

Paper · 2026 · Antonio De Santis, Tommaso Bonetti, Andrea Tocchetti et al.

Summary

The paper asks whether LLMs read between the lines the way people do. The authors define a new task, Implicit Information Extraction (IIE). They build a few-shot prompting pipeline that works with black-box models and turns a context sentence into a two-tier knowledge graph. The pipeline has three stages:

  • extracting entities and (SUBJECT, RELATIONSHIP, OBJECT) triplets, both explicit and implicit, guided by ATOMIC-style inference types;
  • validating and correcting implicit inferences;
  • analysing temporal relations between events.

Graphs produced by two LLMs (a Mistral model and a GPT model) were checked against crowdsourced judgments from more than 300 annotators, using sentences from SocialIQA and COPA.

Overall, humans accepted most model triplets but kept proposing new ones, which shows limited coverage. How strict each side was depended on the data. On socially rich SocialIQA, the models discarded inferences that people considered reasonable. On short, factual COPA, humans became the stricter side. Ordering events in time was the weakest part of the pipeline. The work matters for interpretability and for trustworthy use of LLM-generated text.

Key points

  • Pipeline design: The pipeline is fully automated, model-agnostic and needs no fine-tuning. Nested triplets, inspired by RDF reification, handle subordinate clauses and aspectual verbs. A <NONE> object tag handles unary relations.
  • Human strictness rises on COPA: The share of model triplets that humans marked wrong went from 7.0% to 10.9% for Mistral and from 1.5% to 14.9% for GPT. On COPA, model and human label distributions no longer differed significantly (chi-squared p = 0.1346 for Mistral, p = 0.9407 for GPT).
  • Temporal relations: The models chose "No clear relation" far more often than people.
    • SocialIQA: 33.3% for the models vs 6.7% for humans.
    • COPA: 34.8% for the models overall (GPT 50.0%, Mistral 18.2%) vs 0% for the human consensus.
  • Human consensus: Agreement was above chance in every section (p < 0.001). On SocialIQA it was highest for model error correction (88.7%) and triplet classification (83.2%). The weakest agreement came in the section reviewing corrected inferences (65.1%) and the section comparing event timing (68.5%).
  • Stated limitations: The study covers few examples and only two LLMs. Cross-dataset comparisons may be confounded by differences between the annotator populations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Politecnico di MilanoMilan · Italy5 works · 258 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Antonio De Santis, Tommaso Bonetti, Andrea Tocchetti and Marco Brambilla. 2026. Comparing Human and Large Language Model Interpretation of Implicit Information. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1111

@inproceedings{santis2026comparing,
  title = {{Comparing Human and Large Language Model Interpretation of Implicit Information}},
  author = {Antonio {De Santis}, Tommaso Bonetti, Andrea Tocchetti and Marco Brambilla},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1111},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1111}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F Allen and Patrick J Hayes. 1989. Moments and Points in an Interval-Based Temporal Logic. Computational Intelligence, 5(3):225–238. find
  2. Ali Balali, Masoud Asadpour, and Seyed Hossein Jafari. 2025. COfEE: A Comprehensive Ontology for Event Extraction from Text. Computer Speech & Language, 89:101702. find
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. ACM. find
  4. Creston Brooks, Samuel Eggert, and Denis Peskoff. 2024. The Rise of AI-Generated Content in Wikipedia. Computing Research Repository (CoRR), abs/2410.08044. find
  5. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling The Internal Knowledge-Base of Language Models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1811–1824. Association for Computational Linguistics.
  6. Elias Crum, Antonio De Santis, Manon Ovide, Jiaxin Pan, Alessia Pisu, Nicolas Lazzari, and Sebastian Rudolph. 2024. Enriching ontologies with disjointness axioms using large language models. In Proceedings of the 2nd workshop on Knowledge Base Construction from Pre-Trained Language Models (KBC-LM 20… find
  7. Antonio De Santis, Marco Balduini, Federico De Santis, Andrea Proia, Arsenio Leo, Marco Brambilla, and Emanuele Della Valle. 2025. Integrating large language models and knowledge graphs for extraction and validation of textual test data. In The Semantic Web – ISWC 2024, pages 304–323, Cham. Springer… find
  8. Umberto Eco. 1979. Lector in fabula. La cooperazione interpretativa nei testi narrativi. Bompiani. find
  9. Sefika Efeoglu and Adrian Paschke. 2024. Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks. Computing Research Repository (CoRR). find
  10. John Fields, Kevin Chovanec, and Praveen Madiraju. 2024. A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe? IEEE Access, 12:6518–6531. find
  11. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A Survey of Confidence Estimation and Calibration in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  12. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s…
  13. Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, and 1 others. 2023. A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage. Authorea Preprints, 3. find
  14. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems,…
  15. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, and 79 others. 2024. GPT-4o Sy… find
  16. Uri Katz, Mor Geva, and Jonathan Berant. 2022. Inferring Implicit Relations in Complex Questions with Language Models. pages 2548–2566. find
  17. Timotej Knez and Slavko Žitnik. 2023. Event-Centric Temporal Knowledge Graph Construction: A Survey. Mathematics, 11(23). find
  18. Weixin Liang, Yaohui Zhang, Mihai Codreanu, Jiayu Wang, Hancheng Cao, and James Zou. 2025. The Widespread Adoption of Large Language Model-Assisted Writing Across Society. Computing Research Repository (CoRR), abs/2502.09747. — [arXiv:2502.09747](https://arxiv.org/abs/2502.09747) find
  19. Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D. Manning, and James Y. Zou. 2024. Mapping the Increasing Use of LLMs in Scientific Papers. Computing Research Repository (CoRR). find
  20. Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoyu Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, and 1 others. 2023. Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty. Computing Research Repository (CoRR). find
  21. Mingyu Derek Ma, Jiao Sun, Mu Yang, Huang Kung-Hsiang, Nuan Wen, Shikhar Singh, Rujun Han, and Nanyun Peng. 2021. EventPlus: A Temporal Event Understanding Pipeline. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… find
  22. Mistral AI. 2024. Large Enough. [Accessed: February 26, 2025]. find
  23. Liu Pai, Wenyang Gao, Wenjie Dong, Lin Ai, Ziwei Gong, Songfang Huang, Li Zongsheng, Ehsan Hoque, Julia Hirschberg, and Yue Zhang. 2024. A Survey on Open Information Extraction from Rule-based Model to Large Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa… find
  24. Charles A. Perfetti and Joseph Z. Stafura. 2015. Comprehending Implicit Meanings in Text without Making Inferences, page 1–18. Cambridge University Press. find
  25. James Pustejovsky, Robert Ingria, Roser Saurí, José M. Castaño, Jessica Littman, Robert J. Gaizauskas, Andrea Setzer, Graham Katz, and Inderjeet Mani. 2005. The Specification Language TimeML. In The Language of Time - A Reader, pages 545–558. Oxford University Press. find
  26. Marco Rospocher, Marieke Van Erp, Piek Vossen, Antske Fokkens, Itziar Aldabe, German Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard. 2016. Building Event-Centric Knowledge Graphs from News. Journal of Web Semantics, 37:132–151. find
  27. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019a. Atomic: An Atlas of Machine Commonsense for If-Then Reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Fir… find
  28. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  29. Roser Saurí, Marc Verhagen, and James Pustejovsky. 2006. Annotating and Recognizing Event Modality in Text. In Proceedings of the Nineteenth International Florida Artificial Intelligence Research Society Conference, pages 333–339, Melbourne Beach, Florida, USA. AAAI Press. find
  30. Frank Manola and Eric Miller. 2004. RDF Reification. W3C Recommendation, W3C. find
  31. Pasi Shailendra, Rudra Chandra Ghosh, Rajdeep Kumar, and Nitin Sharma. 2024. Survey of Large Language Models for Answering Questions Across Various Fields. In 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS), volume 1, pages 520–527. find
  32. Mary L McHugh. 2012. Interrater Reliability: The Kappa Statistic. Biochemia Medica, 22(3):276–282. find
  33. Amit Sheth, Swati Padhee, and Amelie Gyrard. 2019. Knowledge Graphs and Knowledge Networks: The Story in Brief. IEEE Internet Computing, 23(4):67– 75. find
  34. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. In Proceedings of the 2022 Conference of the North American Chapter of the Asso…
  35. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the Power of LLMs in Practice: A Survey on Chat-GPT and Beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32. find
  36. Bowen Zhang and Harold Soh. 2024. Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 9820–9836. Association for Computational Linguistics.
  37. Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A. Malin, and Kumar Sricharan. 2023. SAC 3 : Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10,… find
  38. Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. 2023. A Comprehensive Survey on Automatic Knowledge Graph Construction. ACM Computing Surveys, 56(4). find
  39. Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large Language Models for Information Retrieval: A Survey. Computing Research Repository (CoRR).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santis-2026-comparing-human-large-language-model/