Elamur Research⌕
☰
home / papers / 2026 / Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task

Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task

Summary

The paper asks whether prompting LLMs can stand in for surveys of ordinary speakers when deciding what legal text means, for example whether an item counts as covered under an insurance definition. Some judges and legal scholars have proposed this. Earlier studies raised doubts but never collected original human and model responses under the same prompt manipulations, and they did not test reasoning models. The authors adapted 46 insurance-claim vignettes from Waldon et al. (2023). Each vignette came in covered, uncovered and controversial versions and was paired with a positively or negatively framed statement and two answer orders. A representative U.S. sample on Prolific and five LLM configurations answered these items, and coverage decisions were modeled with logistic regression. Humans were unaffected by the irrelevant wording changes, while non-reasoning LLMs shifted their answers with the framing. Turning on reasoning removed significant frame and order effects. However, even reasoning models agreed only moderately with human judgments and sometimes reached conclusions no participant did. The authors therefore advise against using LLMs as substitutes for human subjects in legal interpretation.

Key points

  • Design: 3 versions × 2 frames × 2 answer orders produced 552 distinct item–prompt variants.
  • Human sample: 300 Prolific participants were recruited, and 288 remained after removing those who finished too quickly. Each saw six items, so the typical prompt received three human responses (median).
  • Models tested: Llama-3.3-70B, GPT-4.1 and GPT-5.2 without reasoning, plus GPT-5 Mini and GPT-5.2 with medium-effort reasoning. Each configuration answered all 552 prompts.
  • Analysis: Decisions were analyzed with logistic regression in R, with terms selected by likelihood ratio tests and effects reported as odds ratios with 95% confidence intervals.
  • Findings:
    • Non-reasoning models showed framing effects, while humans and reasoning models did not.
    • All models had at most moderate correlation with human judgments.
    • In some scenarios, all five models denied coverage when no human did.
    • Code is released at github.com/Dawson-Petersen/Sense-and-Sensitivity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.4

@inproceedings{petersen2026sense,
  title = {{Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task}},
  author = {Petersen, Dawson and Purushothama, Abhishek and Schneider, Nathan},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.4},
  url = {https://doi.org/10.18653/v1/2026.conll-main.4}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Richard B Anderson, Jennifer C Crawford, and Michael H Bailey. 2022. Biasing the input: A yoked-scientist demonstration of the distorting effects of optional stopping on bayesian inference. Behavior Research Methods, 54(3):1131–1147. ↗
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. Preprint, arXiv:2204.05862. ↗
  3. Tom Heyman and Geert Heyman. 2019. Can prediction-based distributional semantic models predict typicality? Quarterly Journal of Experimental Psychology, 72(8):2084–2109. ↗
  4. Tom Heyman and Geert Heyman. 2024. The impact of ChatGPT on human data collection: A case study involving typicality norming data. Behavior Research Methods, 56(5):4974–4981. ↗
  5. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics. find our card
  6. Joseph M Hilbe. 2009. Logistic regression models. Chapman and hall/CRC. ↗
  7. David A Hoffman and Yonathan Arbel. 2024. Generative interpretation. New York University Law Review, page 451. find
  8. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. ↗
  9. James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. 2024. Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4):401– 416. ↗
  10. Diana Kornbrot. 2014. Point biserial correlation. Wiley StatsRef: Statistics Reference Online. ↗
  11. D. Kurzon. 2006. Law and language: Overview. In Encyclopedia of Language & Linguistics (Second Edition), pages 728–731. Elsevier, Oxford. ↗
  12. J Martin Bland and Douglas G Altman. 2000. The odds ratio. BMJ, 320(7247):1468. find
  13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pages 611–626, New York, NY, USA. Association for Computing Machinery. find
  14. Benjamin Minhao Chen. 2025. Do linguistic canons matter? Connecticut Law Review (Forthcoming). ↗
  15. Carissa Chen. 2024. The textualist test for large language models. Preprint, Social Science Research Network:5495399. ↗
  16. Thomas R. Lee and Jesse Egbert. 2024. Artificial Meaning? Preprint, Social Science Research Network:4973483. find
  17. Abhishek Purushothama, Junghyun Min, Brandon Waldon, and Nathan Schneider. 2026. Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis. In FAccT ’26: The 2026 ACM Conference on Fairness, Accountability, and Transparency. find
  18. Hannes Leeb and Benedikt M Pötscher. 2009. Model selection. In Handbook of Financial Time Series, pages 889–925. Springer. find
  19. R Core Team. 2024. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  20. Eric Martínez. 2025. Traditional and Computational Canons. Preprint, Social Science Research Network:5155444. ↗
  21. Antonin Scalia and Bryan A. Garner. 2011. Reading Law: The Interpretation of Legal Texts. West Group. find
  22. Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2928– 2949, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  23. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025. OpenAI GPT-5 system card. Preprint, arXiv:2601.03267. ↗ our card
  24. Simon Moss. 2008. Acquiescence bias. find
  25. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations. find
  26. Kevin Newsom. 2024a. Concurring opinion in Snell v. United Specialty Insurance Co. find
  27. Kevin Newsom. 2024b. Concurring opinion in United States v. Deleon. find
  28. Lawrence M Solan and Tammy Gales. 2017. Corpus linguistics as a tool in legal interpretation. Brigham Young University Law Review, 2017(6):1311–1357. find
  29. Aileen Nielsen, Chelse Swoopes, and Elena Glassman. 2025. Law is vulnerable to AI influence; Interface design can help. Preprint, Social Science Research Network:5387231. ↗
  30. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? A case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026. find
  31. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024a. GPT-4 technical report. Preprint, arXiv:2303.08774. ↗ our card
  32. Kevin Tobia. 2020. Testing ordinary meaning. Harvard Law Review, 134:726. find
  33. Kevin Tobia. 2021. The corpus and the courts. The University of Chicago Law Review. find
  34. OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, and 243 others. 2024b. OpenAI o1 system card. Preprint, arXiv:2412.16720. ↗ our card
  35. Kevin Tobia. 2022. Experimental jurisprudence. The University of Chicago Law Review, 89(3):735–802. find
  36. Brandon Waldon, Madigan Brodsky, Megan Ma, and Judith Degen. 2023. Predicting consensus in legal document interpretation. Proceedings of the Annual Meeting of the Cognitive Science Society, 45(45). find
  37. Brandon Waldon, Nathan Schneider, Ethan Wilcox, Amir Zeldes, and Kevin Tobia. 2025. Large language models for legal interpretation? Don’t take their word for it. The Georgetown Law Review, 114. find
  38. Andrea Pedrotti, Giulia Rambelli, Caterina Villani, and Marianna Bolognesi. 2025. How humans and LLMs organize conceptual knowledge: Exploring subordinate categories in Italian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4464–4482. ↗
  39. Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. 2024. From decoding to meta-generation: Inference-time algorithms for large language models. Preprint, arXiv:2406.16838. ↗
  40. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with GPT-4. Preprint, arXiv:2304.03277. ↗ our card
  41. Daniel Wilf-Townsend and Kevin Tobia. 2025. Generative AI and courts in the United States. Preprint, Social Science Research Network:5243402. ↗
  42. Prolific. 2025. What are representative samples on prolific. *Figure 5: A reference example question from the survey. This is not an exact replication of how it was shown to the subjects in the interactive web interface.* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/petersen-2026-sense-sensitivity-reasoning-models-are/