Elamur Research⌕
☰
home / papers / 2024 / The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English

The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English

Paper · 2024 · Tom S Juzek

Summary

Syntacticians usually judge sentences against their own formal theories (grammaticality), while ordinary speakers judge how natural a sentence feels (acceptability). Large public resources such as CoLA contain only the expert labels, and public acceptability collections have stayed small, at a few hundred items. This preview paper releases 1,000 English example sentences from the syntax literature: half sampled from CoLA, which draws on textbooks, and half from Linguistic Inquiry items, so that current research is represented. Each sentence keeps its original expert label and gets crowdsourced ratings from native American English speakers on a 7-point scale, with an average of more than 30 ratings per item under strict quality controls. Expert and lay binary labels agree in roughly 83% of cases. Agreement is much weaker for sentences experts call ill-formed, and many items fall in the middle of the scale. Preliminary experiments suggest ML models predict acceptability considerably better than grammaticality. The resource is the largest public dataset of its kind and is planned to grow to about 15,000 items.

Key points

  • The dataset has 500 sentences sampled from CoLA (textbook condition) and 500 from the Juzek and Häussler (2020) Linguistic Inquiry sample (journal condition). The grammatical share is 71.4% in the textbook condition and 67.4% in the journal condition. Items are singletons, not minimal pairs.
  • Ratings were collected on Prolific from 597 participants, paid about $15/hr. Each participant rated 77 items: 64 critical items, 4 calibration items, 4 language-proficiency checks and 5 attention checks. The platform issued response-time warnings and excluded participants who failed the checks.
  • In total there are 34,490 judgments. Item means span almost the whole scale, from 1.35 to 6.98. Earlier public sets held about 400 items (Lau et al., 2017) and about 200 items (Warstadt et al., 2019).
  • Each entry has these fields:
    • the expert grammaticality label;
    • the raw mean rating;
    • the rating normalized to 0–1;
    • a binary label using a 0.5 cutoff, with exact ties settled by the grammaticality label;
    • the source (textbook or journal).
  • Convergence between expert and lay labels:
    • Overall agreement is 83.3%, with 85.8% in the textbook condition and 80.8% in the journal condition.
    • Agreement is 89.3% for grammatical items but only 69.6% for ungrammatical ones.
    • For example, "*John perfectly rolled the ball" averaged 6.22.

Interesting ideas

  • I1. The dataset stores the expert formal label and the lay judgment side by side, so cases where they disagree become visible rather than being folded into one target.
  • I2. Quality is controlled inside the task itself, through calibration anchors at both scale ends, planted proficiency and attention items, and speed warnings. Each new rater also gets the items with the fewest ratings so far.
  • I3. The dataset keeps a graded score and derives the binary label from it with a fixed tie-breaking rule, instead of collecting only yes/no answers. This preserves "in-between" cases for later analysis.
  • I4. Agreement on "bad" items is much weaker than on "good" ones, which means expert negatives are the less trustworthy half of the labels.
  • I5. Results are reported separately by provenance (textbook vs. journal), and this exposes differences that a single overall figure would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Sentences that theory rejects but speakers accept (e.g., bare-date topics) make a good robustness suite, since the UD parser should still produce sensible trees for them.
  • I2 (learn): The same pattern fits LLM teacher tiers and witness parsers. Seed known-answer control sentences to detect unreliable teachers, and route new sentences to whichever has the fewest votes so far.
  • I3 (learn): Silver-data agreement could be kept as a graded score, with binarization applied only at evaluation time by a deterministic, documented tie rule.
  • I4 (learn): Negative controls made from supposedly ungrammatical perturbations should be spot-checked, because many may be acceptable English and would wrongly penalize a correct parse.
  • I5 (answer): Evaluation of school tests, word problems and fable QA could also be broken down by text source, so that drops on newer or less canonical material show up.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Florida State UniversityTallahassee · United States8 works · 363 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Tom S Juzek. 2024. The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{juzek2024syntactic,
  title = {{The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English}},
  author = {Tom S Juzek},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.17605/osf.io/5e8kz},
  url = {https://doi.org/10.17605/osf.io/5e8kz}
}

References

59 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adger. 2003. Core syntax: A minimalist approach. Oxford University Press. find
  2. Markus Bader and Jana Häussler. 2010. Toward a model of grammaticality judgments. Journal of Linguistics, 46(2):273–330. ↗
  3. Mark Baltin and Chris Collins. 1991. The handbook of contemporary syntactic theory. John Wiley & Sons. find
  4. David Basilico. 2003. The topic of small clauses. Linguistic Inquiry, 34(1):1–35. ↗
  5. Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. 2020. How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International conference on spoken language translation, pages 280–290. find
  6. Rui P Chaves. 2020. What don’t rnn language models learn about filler-gap dependencies? Proceedings of the Society for Computation in Linguistics, 3(1):20–30. ↗
  7. Rui P Chaves and Stephanie N Richter. 2021. Look at that! bert can be easily distracted from paying attention to morphosyntax. Proceedings of the Society for Computation in Linguistics, 4(1):28– 38. ↗
  8. Zhong Chen, Yuhang Xu, and Zhiguo Xie. 2020. Assessing introspective linguistic judgments quantitatively: the case of the syntax of chinese. Journal of East Asian Linguistics, 29:311–336. ↗
  9. Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):1–13. ↗
  10. Nancy Chinchor. 1992. Muc-4 evaluation metrics. In Proceedings of the 4th conference on Message understanding-MUC4’92. Association for Computational Linguistics. find
  11. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge (MA). ↗
  12. Noam Chomsky. 1975. The Logical Structure of Linguistic Theory. Springer US. find
  13. Noam Chomsky, George Armitage Miller, R Luce, R Bush, and E Galanter. 1963. Introduction to the formal analysis of natural languages. 1963, pages 269–321. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  15. Urtzi Etxeberria, Susagna Tubau, Viviane Déprez, Joan Borràs-Comes, and M Teresa Espinal. 2018. Relating (un) acceptability to interpretation. experimental investigations on negation. Frontiers in psychology, 8:274247. ↗
  16. Gisbert Fanselow, Jana Häussler, and Thomas Weskott. 2019. Who cares what who prefers? a study in judgment differences between syntacticians and non-syntacticians. Grammatical approaches to language processing: Essays in honor of Lyn Frazier, pages 261–274. ↗
  17. Sam Featherston. 2005. The decathlon model of empirical syntax in: Reis, marga, kepser, stephan stephan (eds.), linguistic evidence. empirical, theoretical and computational perspectives. find
  18. Ingo Feldhausen and Sebastian Buchczyk. 2021. Revisiting subjunctive obviation in french: a formal acceptability judgment study. Glossa: a journal of general linguistics. 2021; 6 (1): 59. ↗
  19. Elaine Francis. 2021. Gradient acceptability and linguistic theory, volume 11. Oxford University Press. find
  20. Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive science, 44(3):e12814. ↗
  21. Edward Gibson, Steven T Piantadosi, and Evelina Fedorenko. 2013. Quantitative methods in syntax/semantics research: A response to sprouse and almeida (2013). Language and Cognitive processes, 28(3):229–240. ↗
  22. Edward Gibson et al. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. Image, language, brain, 2000:95– 126. ↗
  23. Dirk Goldhahn, Thomas Eckart, Uwe Quasthoff, et al. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In LREC, volume 29, pages 31–43.
  24. Carolina González and Lara Reglero. 2024. Intonation correlates of canonical and non-canonical wh-in-situ questions in spanish. Journal of Linguistics, 60(1):29–50. ↗
  25. Carolina González, Lara Reglero, Barbara E Bullock, Cinzia Russi, and Almeida Jacqueline Toribio. 2022. Prosodic correlates of mirative and new information focus in spanish wh-in-situ questions. A half century of Romance linguistics, page 269. ↗
  26. Alexander Grosu and Julia Horvath. 2006. Reply to bhatt and pancheva’s “late merger of degree clauses”: The irrelevance of (non) conservativity. Linguistic Inquiry, 37(3):457–483. find
  27. Jana Häussler and Tom Juzek. 2020. Linguistic intuitions and the puzzle of gradience. Linguistic intuitions: Evidence and method, pages 233– 254. ↗
  28. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138.
  29. Bradley Hoot and Shane Ebert. 2021. The that-trace effect: Evidence from spanish–english code-switching. Languages, 6(4):189. ↗
  30. Tom S Juzek and Jana Häussler. 2020. Data convergence in syntactic theory and the role of sentence pairs. Zeitschrift für Sprachwissenschaft, 39(2):109–147. ↗
  31. Jong-Bok Kim and Peter Sells. 2008. English syntax: An introduction. CSLI publications Stanford, CA. find
  32. Marie-Pauline Krielke. 2024. Cross-linguistic dependency length minimization in scientific language: Syntactic complexity reduction in english and german in the late modern period. Languages in Contrast, 24(1):133–163. ↗
  33. Irene Lami and Joost van de Weijer. 2022. Compound-internal anaphora: evidence from acceptability judgements on italian argumental compounds. Morphology, 32(4):359–388. ↗
  34. Idan Landau. 2007. Epp extensions. Linguistic Inquiry, 38(3):485–523. find
  35. Steven Langsford, Amy Perfors, Andrew T Hendrickson, Lauren A Kennedy, and Danielle J Navarro. 2018. Quantifying sentence acceptability measures: Reliability, bias, and variability. Glossa: a journal of general linguistics, 3(1):1– 34. ↗
  36. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive science, 41(5):1202–1241. ↗
  37. Paulina Łęska-Bayraktar and Sylwiusz Żychliński. 2023. A panoramic view of acceptability judgments in polish generative linguistics. Poznan Studies in Contemporary Linguistics, 59(4):705– 740. ↗
  38. Haitao Liu. 2010. Dependency direction as a means of word-order typology: A method based on dependency treebanks. Lingua, 120(6):1567– 1578. ↗
  39. Kyle Mahowald, Jeremy Hartman, Peter Graff, and Edward Gibson. 2016. Snap judgments: A small n acceptability paradigm (snap) for linguistic acceptability judgments. Language, pages 619– 635. ↗
  40. Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553. ↗
  41. Samuel Schindler, Anna Drożdżowicz, and Karen Brøcker. 2020. Linguistic intuitions: Evidence and method. Oxford University Press. find
  42. Gert-Jan Schoenmakers. 2023. Linguistic judgments in 3d: the aesthetic quality, linguistic acceptability, and surface probability of stigmatized and non-stigmatized variation. Linguistics, 61(3):779–824. ↗
  43. Carson T. Schütze. 1996. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press, Chicago, IL. find
  44. Carson T. Schütze. 2020. Acceptability ratings cannot be taken at face value. In Samuel Schindler, Anna Drożdżowicz, and Karen Brøcker, editors, Linguistic Intuitions: Evidence and Method, chapter 11, pages 189–214. Oxford University Press, Oxford. ↗
  45. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. ↗
  46. Dominique Sportiche, Hilda Koopman, and Edward Stabler. 2013. An introduction to syntactic analysis and theory. John Wiley & Sons. find
  47. Jon Sprouse, Carson T Schütze, and Diogo Almeida. 2013. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001–2010. Lingua, 134:219–248. ↗
  48. Kevin Tang and Jason A Shaw. 2021. Prosody leaks into the memories of words. Cognition, 210:104601. ↗
  49. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316. ↗
  50. Lucien Tesnière. 2015. Elements of structural syntax. John Benjamins Publishing Company. ↗
  51. Etxeberria Urtzi, Tubau Susagna, Joan Borràs-Comes, and Espinal M Teresa. 2022. Polarity items in basque. Natural Language & Linguistic Theory, 40(2):485–504. ↗
  52. C.J. Van Rijsbergen. 1979. Information Retrieval. Butterworths. find
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  54. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34. ↗
  55. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. ↗
  56. Thomas Wasow. 2007. Gradient data and gradient grammars. In Proceedings from the Annual Meeting of the Chicago Linguistic Society, volume 43, pages 255–271. Chicago Linguistic Society. find
  57. Thomas Wasow and Jennifer Arnold. 2005. Intuitions in linguistic argumentation. Lingua, 115(11):1481–1496. ↗
  58. Thomas Wasow, Roger Levy, Robin Melnick, Hanzhi Zhu, and Tom Juzek. 2015. Processing, prosody, and optional to. Explicit and Implicit Prosody in Sentence Processing: Studies in Honor of Janet Dean Fodor, pages 133–158. ↗
  59. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/juzek-2024-syntactic-acceptability-dataset-preview-resource/