Elamur Research⌕
☰
home / papers / 2026 / Linguistically-Informed Evaluation of LLMs on Acceptability Judgments in a Forced-Choice Paradigm

Linguistically-Informed Evaluation of LLMs on Acceptability Judgments in a Forced-Choice Paradigm

Paper · 2026 · Ziyue Liu and Nils Reiter

Summary

Whether LLMs have human-like grammatical knowledge is still debated. Earlier replications of human acceptability studies used a single model and one fixed prompt, and they did not apply a linguistic taxonomy. Liu and Reiter revisit the forced-choice (FC) paradigm with the stimuli of Sprouse et al. (2013), which come from Linguistic Inquiry. They relabel each phenomenon with BLiMP categories and test GPT-3.5, GPT-4o, GPT-OSS, Llama and Claude against judgments from 307 native English speakers.

The comparison covers mean accuracy and also the full distribution of responses, measured with the Wasserstein distance. A second experiment varies the prompt: linguist vs. layperson persona, detailed vs. brief instructions, and user-only vs. user-system messages.

Most models match or exceed human accuracy. However, several hit ceiling effects that erase the item-level variation seen in people. Models do well on agreement phenomena but are weaker and less stable on binding, ellipsis and quantifiers. Prompt changes have little effect.

The study shows that agreement on aggregate scores can hide qualitatively different behaviour. This supports linguistically grounded evaluation of LLMs.

Key points

  • Data: Sprouse et al. (2013) items from Linguistic Inquiry (2001–2010). After removing two duplicates, 148 phenomena remain, giving 1,184 sentence pairs. The phenomena were classified into BLiMP types by ChatGPT-5, Claude Sonnet-4.5 and a PhD student.
  • Setup: Each item was queried 50 times per model at temperature 0.5, using the "linguist" prompt from Qiu et al. (2025). Models were accessed via OpenRouter, and GPT-OSS-120B via the University of Cologne.
  • Distribution match (Wasserstein distance to humans):
    • Llama 0.019 (closest)
    • GPT-OSS 0.021
    • GPT-3.5 0.023
    • GPT-4o 0.038
    • Claude 0.042
  • Phenomenon results:
    • Human accuracy averages about 0.93, with ellipsis hardest for people (about 0.80).
    • Binding is the weakest category across all models.
    • On island phenomena, a few models (Claude, GPT-4o, GPT-OSS) score higher than the human participants.
  • Prompting: Varying persona, instruction length and message format barely changes performance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CologneCologne · Germany9 works · 407 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Ziyue Liu and Nils Reiter. 2026. Linguistically-Informed Evaluation of LLMs on Acceptability Judgments in a Forced-Choice Paradigm. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.103

@inproceedings{liu2026linguistically,
  title = {{Linguistically-Informed Evaluation of LLMs on Acceptability Judgments in a Forced-Choice Paradigm}},
  author = {Ziyue Liu and Nils Reiter},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.103},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.103}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel Amouyal, Aya Meltzer-Asscher, and Jonathan Berant. 2024. Large language models for psycholinguistic plausibility pretesting. In Findings of the Association for Computational Linguistics: EACL 2024, pages 166–181. find
  2. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  3. Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. In Philosophy, language, and artificial intelligence: Resources for processing natural language, pages 241–301. Springer. find
  4. Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5185–5198.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Zhenguang Cai, Xufeng Duan, David Haslett, Shuqi Wang, and Martin Pickering. 2024. Do large language models resemble humans in language use? In Proceedings of the workshop on cognitive modeling and computational linguistics, pages 37–56.
  7. Noam Chomsky. 1981. Lectures on government and binding, foris, dordrecht. ChomskyLectures on Government and Binding1981. find
  8. Noam Chomsky. 2014. Aspects of the Theory of Syntax. 11. MIT press. find
  9. Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons. find
  10. Wayne Cowart. 1997. Experimental syntax. Sage. find
  11. Ruixiang Cui, Daniel Hershcovich, and Anders Søgaard. 2022. Generalized quantifiers as a source of error in multilingual nlu benchmarks. In Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 4875– 4893… find
  12. Vittoria Dentella, Fritz Günther, and Evelina Leivada. 2023. Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. find
  15. Rena Wei Gao, Xuetong Wu, Tatsuki Kuribayashi, Mingrui Ye, Siya Qi, Carsten Roever, Yuanxing Liu, Zheng Yuan, and Jey Han Lau. 2025. Can llms simulate l2-english dialogue? an information-theoretic analysis of l1-dependent biases. In Proceedings of the 63rd Annual Meeting of the Association for Compu… find
  16. Akshat Gupta. 2023. Probing quantifier comprehension in large language models: Another example of inverse scaling. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 56–64. find
  17. Jacqueline Harding, William DAlessandro, NG Laskowski, and Robert Long. 2024. Ai language models cannot replace human research participants. Ai & Society, 39(5):2603–2605. find
  18. Irene Heim and Angelika Kratzer. 1998. Semantics in generative grammar. find
  19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. 2024. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121. find
  20. Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2025. How to make the most of llms grammatical knowledge for acceptability judgments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computat… find
  21. Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition. a reply to piantadosi (2023). Manuscript. Tel Aviv University. url: https://lingbuzz.net/lingbuzz/007190. ↗
  22. S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. find
  23. Idan Landau. 2013. Control in generative grammar: A research companion. Cambridge University Press. find
  24. Zhicheng Lin. 2025. Large language models as psychological simulators: A methodological guide. arXiv preprint arXiv:2506.16702. — [arXiv:2506.16702](https://arxiv.org/abs/2506.16702) find
  25. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2025. A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5):1–72.
  26. Barbara BH Partee, Alice G Ter Meulen, and Robert Wall. 2012. Mathematical methods in linguistics, volume 30. Springer Science & Business Media. find
  27. Axel Pichler, Janis Pagel, and Nils Reiter. 2025. Evaluating llm-prompting for sequence labeling tasks in computational literary studies. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025),… find
  28. Zhuang Qiu, Xufeng Duan, and Zhenguang G Cai. 2025. Grammaticality representation in chatgpt as compared to linguists and laypeople. Humanities and Social Sciences Communications, 12(1):1–15. find
  29. Carson T Schütze, Jon Sprouse, Robert J Podesva, and Devyani Sharma. 2014. Research methods in linguistics. Cambridge University Press Cambridge. find
  30. Dominique Sportiche. 1988. A theory of floating quantifiers and its corollaries for constituent structure. Linguistic inquiry, 19(3):425–449. find
  31. Jon Sprouse, Carson T Schütze, and Diogo Almeida. 2013. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001–2010. Lingua, 134:219–248. find
  32. Hans Stadthagen-González, Luis López, M Carmen Parafita Couto, and C Alejandro Párraga. 2018. Using two-alternative forced choice tasks and thur-stones law of comparative judgments for code-switching research. Linguistic Approaches to Bilingualism, 8(1):67–97. find
  33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  34. Leonid Nisonovich Vaserstein. 1969. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64–72. find
  35. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392.
  36. Robert May. 1985. Logical form: Its structure and derivation, volume 12. MIT press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-linguistically-informed-evaluation-llms-acceptability/