Elamur Research⌕
☰
home / papers / 2026 / Open-access Dataset on Acceptability Ratings of Korean Clausal Constructions by Humans and GPT Models

Open-access Dataset on Acceptability Ratings of Korean Clausal Constructions by Humans and GPT Models

Summary

Most work on whether language models mirror human linguistic judgements targets English. Korean has had no openly available set of human sentence ratings comparable to SyntaxGym. The authors address this with a released resource covering three Korean clausal phenomena where morphosyntax meets semantics:

  • the dative case alternation,
  • active versus passive voice, crossed with theme animacy,
  • licensing of the NPI amwuto by negation under the clausemate condition.

They built controlled sentence sets and gathered acceptability judgements from adult native speakers in their twenties. They paired these with sentence-level surprisal from GPT-2 and GPT-Trinity, plus prompted ratings with short justifications from ChatGPT. This tests a linking hypothesis: can model outputs stand in for human judgement?

Alignment turned out to depend on both model and construction. Models followed structurally driven contrasts such as the dative alternation more closely than semantically driven ones like passive animacy or NPI licensing. Surprisal correlated only weakly with human ratings. The dataset and code give a starting point for evaluating LLMs beyond English.

Key points

  • The stimuli comprise 224 sentences: 96 dative (48 Dative–Accusative, 48 Accusative–Accusative), 64 active/passive (2×2 animacy × voice, 16 per cell) and 64 NPI (2×2 NPI location × negation location, 16 per cell). Ten native speakers normed the materials before the main study.
  • Models compared with human ratings: GPT-2 and GPT-Trinity (sentence surprisal) and ChatGPT (prompted ratings with one-sentence rationales).
  • ChatGPT reproduced the main human contrasts. It gave Dative–Accusative a mean of about 4.56 versus 1.92 for Accusative–Accusative, and it rated grammatical NPI conditions above ungrammatical ones. It showed only a weak animacy effect within passives.
  • Correlations with human ratings were small and construction-specific:
    • Dative–Accusative: ChatGPT r ≈ 0.34, GPT-2 r ≈ 0.12.
    • Passives: no reliable surprisal correlations.
    • NPI items: only scattered GPT-Trinity effects.
  • All data and code are on OSF (doi:10.17605/OSF.IO/RQZU3). Stated limitations are the modest size and the variability between ChatGPT prompters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2icd7h29b849

@inproceedings{shin2026open,
  title = {{Open-access Dataset on Acceptability Ratings of Korean Clausal Constructions by Humans and GPT Models}},
  author = {Shin, Gyu-Ho and Lee, Soo-Hwan and Lee, Chanyoung},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2icd7h29b849},
  url = {https://doi.org/10.63317/2icd7h29b849}
}

References

70 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Ambridge, R. Maitreyee, T. Tatsumi, L. Doherty, S. Zicherman, P. M. Pedro, C. Bannard, S. Samanta, S. McCauley, I. Arnon, D. Bekman, A. Efrati, R. Berman, B. Narasimhan, D. M. Sharma, R. B. Nair, K. Fukumura, S. Campbell, C. Pye, S. F. C. Pixabaj, M. M. Paliz, and M. J. Mendoza. 2020. The crossli… find
  2. D. J. Barr, R. Levy, C. Scheepers, and H. J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. find
  3. D. Bates, M. Mächler, B. Bolker, and S. Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1–48. find our card
  4. D. E. Blasi, J. Henrich, E. Adamou, D. Kemmerer, and A. Majid. 2022. Over-reliance on English hinders cognitive science. Trends in Cognitive Sciences, 26(12):1153–1170. find
  5. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find
  6. F. Chang. 2009. Learning to order words: A connectionist model of heavy NP shift and accessibility effects in Japanese and English. Journal of Memory and Language, 61(3):374–397. find
  7. T. A. Chang and B. K. Bergen. 2024. Language model behavior: A comprehensive survey. Computational Linguistics, pages 1–58. find
  8. Y. J. Cho and M. G. Jeon. 2015. hankwuke swuy-ongseng phantanuy silhempangpeplon pikyo yenkwu [a comparative study of acceptability judgment collection methods in Korean]. The Journal of Linguistics Science, 72:397–416. find
  9. H. S. Choe. 1988. Restructuring Paramters and Complex Pedicates - A Transformational Approach. Ph.D. thesis, Massachusetts Institute of Technology. find
  10. M. Choo and H-Y. Kwak. 2008. Using Korean: A Guide to Contemporary Usage. Cambridge University Press, Cambridge. find
  11. R. H. B. Christensen. 2023. Ordinal-Regression Models for Ordinal Data. R package version 2023.12-4. find
  12. P. Contreras Kallens, R. D. Kristensen-McLachlan, and M. H. Christiansen. 2023. Large language models demonstrate the potential of statistical learning in language. Cognitive Science, 47(3):e13256. find
  13. N. Cuneo, N. Graves, S. Rakshit, and A. Goldberg. 2025. For GPT-4 as with humans: Information structure predicts acceptability of long-distance dependencies. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 47. find
  14. A. G. de Varda, M. Marelli, and S. Amenta. 2024. Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data. Behavior Research Methods, 56(5):5190–5213. find
  15. V. Dentella, F. Günther, and E. Leivada. 2023. Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120. find
  16. J. Gauthier, J. Hu, E. Wilcox, P. Qian, and R. Levy. 2020. Syntaxgym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. Association for Computational Linguisti… find our card
  17. A. Goldstein, Z. Zada, E. Buchnik, M. Schain, A. Price, B. Aubrey, S. A. Nastase, A. Feder, D. Emanuel, A. Cohen, A. Jansen, H. Gazula, G. Choe, A. Rao, C. Kim, C. Casto, L. Fanda, W. Doyle, D. Friedman, P. Dugan, L. Melloni, R. Reichart, S. Devore, A. Flinker, L. Hasenfratz, O. Levy, A. Hassidim, M… find
  18. M. Haspelmath. 1990. The grammaticization of passive morphology. Studies in Language, 14(1):25– 72. find
  19. M. Haspelmath. 2015. Ditransitive constructions. Annual Review of Linguistics, 1(1):19–41. find
  20. R. D. Hawkins, T. Yamakoshi, T. L. Griffiths, and A. E. Goldberg. 2020. Investigating representations of verb bias in neural language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4653–4663. Association for Computational Linguistics. find
  21. E. A. Hosseini, M. Schrimpf, Y. Zhang, S. Bowman, N. Zaslavsky, and E. Fedorenko. 2022. Artificial neural network language models align neurally and behaviorally with humans even after a developmentally realistic amount of training. bioRxiv. find
  22. J. Hu, J. Gauthier, P. Qian, E. Wilcox, and R. P. Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Association for Computational Linguistics, pages 1725–1744. Association for Computational Linguistics. find
  23. J. Hu and R. Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics. find our card
  24. K. J. Huang, S. Arehalli, M. Kugemoto, C. Muxica, G. Prasad, B. Dillon, and T. Linzen. 2024. Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty. Journal of Memory and Language, 137:104510. find
  25. C. R. Jones and B. Bergen. 2024. Does word knowledge account for the effect of world knowledge on pronoun interpretation? Language and Cognition, pages 1–32. find
  26. H. Kim and G-H. Shin. 2022. Effects of verb and construction frequency in sentence comprehension: A case of dative construction in Korean. Functions of Language, 29(3):274–299. find
  27. H. Kim, G-H. Shin, and H. Hwang. 2020. Integration of verbal and constructional information in the second language processing of English dative constructions. Studies in Second Language Acquisition, 42(4):825–847. find
  28. S. Kuno. 1998. Negative polarity items in Korean and English. Cornell East Asia Series, 98:87– 132. find
  29. S. Lee and M. Vu. 2024. The effects of distance on NPI illusive effects in BERT. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9443– 9457. find
  30. S-H. Lee. 2024. Language model performance on English control constructions and its implications. Journal of Linguistic Science, pages 245–278. find
  31. S-H. Lee and S. Schuster. 2022. Can language models capture syntactic associations without surface cues? a case study of reflexive anaphor licensing in English control constructions. In Proceedings of the Society for Computation in Linguistics 2022, pages 206–211. find our card
  32. S-H. Lee and S. Wang. 2023. Do language models know how to be polite? Society for Computation in Linguistics, 6(1):375–378. find
  33. Y-h. Lee. 2014. Semantic relations and multiple case constructions: an experimental approach. Linguistic Research, 31(2):213–247. find
  34. R. Lenth. 2025. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.11.1. find
  35. R. Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  36. R. Marvin and T. Linzen. 2018. Targeted syntactic evaluation of language models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  37. R. Marvin and T. Linzen. 2019. Targeted syntactic evaluation of language models. In Proceedings of the Society for Computation in Linguistics, volume 2, pages 373–374. find
  38. S. Nakagawa and H. Schielzeth. 2013. A general and simple method for obtaining R 2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2):133–142. find
  39. B. D. Oh, C. Clark, and W. Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5:777963. find
  40. B. D. Oh and W. Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. find our card
  41. S-H. Park and E. Yi. 2021. Perception-production asymmetry for Korean double accusative ditransitives. Linguistic Research, 38(1):27–52. find
  42. T. Park. 2021. Study on the frequency and causes of the passive in English and Korean in the gospel of john. The Journal of Linguistics Science, 98:195–213. find
  43. A. Parrish, W. Huang, O. Agha, S-H. Lee, N. Nangia, A. Warstadt, K. Aggarwal, E. Allaway, T. Linzen, and S. R. Bowman. 2021a. Does putting a linguist in the loop improve NLU data collection? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4886–4901, Punta Cana, Domini… find
  44. A. Parrish, S. Schuster, A. Warstadt, O. Agha, S-H. Lee, Z. Zhao, S. R. Bowman, and T. Linzen. 2021b. NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 349–366, Online. Association for Computation… find our card
  45. R Core Team. 2025. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  46. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  47. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  48. M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118. find
  49. C. Shain, C. Meister, T. Pimentel, R. Cotterell, and R. Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  50. G-H. Shin and S. Mun. 2023. Korean-speaking children’s constructional knowledge about a transitive event: Corpus analysis and Bayesian modelling. Journal of Child Language, 50(2):311– 337. find
  51. G-H. Shin and S. Mun. 2025. Modelling child comprehension: A case of suffixal passive construction in Korean. Computer Speech & Language, 90:101701. find
  52. J-A. Shin and K. Christianson. 2009. Syntactic processing in Korean–English bilingual production: Evidence from cross-linguistic structural priming. Cognition, 112(1):175–180. find
  53. S-i. Shin. 2016. A study on the functions of eul/reul through examining double accusative constructions: focusing on transitivity. URIMALGEUL: The Korean Language and Literature, 68:1–35. find
  54. U. Shin, E. Yi, and S. Song. 2023. Investigating a neural language model’s replicability of psycholinguistic experiments: A case study of NPI licensing. Frontiers in Psychology, 14:937656. find
  55. A. Siewierska. 2013. Passive constructions. In M. S. Dryer and M. Haspelmath, editors, WALS Online (v2020.3). Max Planck Institute for Evolutionary Anthropology. find
  56. H. M. Sohn. 1999. The Korean Language. Cambridge University Press, Cambridge. find
  57. A. Srivastava, A. Rastogi, A. Rao, A. A. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. find our card
  58. K. Sun and R. Wang. 2025. Computational sentence-level metrics of reading speed and its ramifications for sentence comprehension. Cognitive Science, 49(7):e70092. find
  59. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 31, pages 5998–6008. Curran Associates, Inc. find
  60. A. Warstadt and S. R. Bowman. 2020. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Conference of the Cognitive Science Society, pages 1737–1743. Cognitive Science Society. find
  61. A. Warstadt, Y. Cao, I. Grosu, W. Peng, H. Blix, Y. Nie, A. Alsop, S. Bordia, H. Liu, A. Parrish, S-F. Wang, J. Phang, A. Mohananey, P. M. Htut, P. Jeretič, and S. R. Bowman. 2019a. Investigating BERT’s knowledge of language: Five analysis methods with NPIs. In Proceedings of the 2019 Conference on… find our card
  62. A. Warstadt, A. Singh, and S. R. Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  63. E. Wilcox, R. Levy, T. Morita, and R. Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221. Association for Computational Linguistics. find our card
  64. E. Wilcox, P. Qian, R. Futrell, M. Ballesteros, and R. Levy. 2019. Structural supervision improves learning of non-local grammatical dependencies. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find
  65. I. H. Woo. 1997. wulimal phitong yenkwu [Study on a passive voice in Korean]. Hankwukmwun-hwasa, Seoul. find
  66. W. Xu, J. Chon, T. Liu, and R. Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721. Association for Computational Linguistics. find
  67. J. Yeon. 2003. Korean Grammatical Constructions: Their Form and Meaning. Saffron Books, London. find
  68. M. H. Yoo, J. Kim, and S. Song. 2025. Multilingual capabilities of GPT: A study of structural ambiguity. PLoS One, 20(7):e0326943. find
  69. J. H-S. Yoon. 2015. Double nominative and double accusative constructions. In L. Brown and J. Yeon, editors, The Handbook of Korean Linguistics, pages 79–97. John Wiley Sons, Oxford. find
  70. X. Zhou, D. Chen, S. Cahyawijaya, X. Duan, and Z. Cai. 2025. Linguistic minimal pairs elicit linguistic similarity in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6866–6888. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shin-2026-open-access-dataset-acceptability-ratings/