Elamur Research⌕
☰
home / papers / 2022 / The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

Paper · 2022 · Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal et al.

Summary

The paper asks whether pretrained language models grasp both the form and the meaning of one well-known Construction Grammar (CxG) construction, the English comparative correlative (CC), as in "the more, the merrier". The authors studied BERT, RoBERTa and DeBERTa in two parts.

  • Syntax: they trained logistic-regression probes on layer embeddings to separate CC instances from non-instances. The data came from two sources. One was a context-free grammar that generates approximate minimal pairs, where word order alone decides CC status. The other was a set of manually labelled C4 sentences that either are CCs or closely resemble them.
  • Semantics: they built zero-shot scenarios in which a model is given a CC statement and must apply it. Calibration was used to counter prompting biases.

All three models recognise the construction well, from the third layer onward. However, none of them uses its meaning above chance level. This gap between recognising the structure and understanding it shows that strong surface syntax in PLMs does not guarantee usable semantics.

Key points

  • Models probed: BERT, RoBERTa and DeBERTa, with DeBERTa XXL among the three largest models examined in the calibration analysis.
  • Syntactic probing on CFG-generated pairs separates positive from negative CC instances starting at layer 3 for every model.
  • A diverse set of C4 sentences was labelled by hand, covering true CCs and close look-alikes. The probes still separated these classes very well.
  • In the semantic application test, accuracy stays around chance even after calibrating for recency bias and vocabulary bias. For DeBERTa XXL, scores fall roughly between 47 and 55.
  • In the CC, "the" acts as a degree word or fixed material, not as a determiner of a noun phrase.
  • Data and code are released at github.com/LeonieWeissweiler/ComparativeCorrelative.

Interesting ideas

  • I1. When true minimal pairs are impossible, a small CFG can produce near-minimal pairs in which only word order switches the construction on or off, isolating it from lexical confounds.
  • I2. Hard negatives that are mined from a corpus and hand-labelled can be used, meaning sentences that look like the target pattern but are not instances of it.
  • I3. Testing whether a construction is recognised separately from whether its meaning is applied shows that a system can know the form yet fail to use the content.
  • I4. Before trusting zero-shot judgments, calibrate them against known biases such as preferring the most recent option or favouring certain words.
  • I5. Function words can take construction-specific roles, as when the CC "the" works as a degree marker and not as a determiner.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): generate CC and non-CC pairs to check that the perceptron parser and gparse give consistent, distinct analyses of the paired "the + comparative" clauses.
  • I2 (learn): use such look-alikes as negative controls when evaluating silver data and repair rules, so a rule that fires on surface cues alone gets caught.
  • I3 (frames): map the CC to an explicit correlation frame, independent and dependent variable, and then check in answer-stage tests that the frame actually changes conclusions.
  • I4 (learn): apply similar calibration to the LLM teacher tiers so that their labels reflect the content rather than option order or word preference.
  • I5 (tokens): let the tagger and lemma layer flag this "the" by its construction, so later stages do not treat it as an ordinary determiner.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksSyntax & grammar formalisms · 214 worksInterpretability · 130 works

Cite

Cite as Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal and Hinrich Schütze. 2022. The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.746

@inproceedings{weissweiler2022better,
  title = {{The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative}},
  author = {Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal and Hinrich Schütze},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.746},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.746}
}

References

51 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne Abeillé and Robert D Borsley. 2008. Comparative correlatives and parameters. Lingua, 118(8):1139– 1157. ↗
  2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada. Association for Computational Linguistics.
  3. Giulia ML Bencini and Adele E Goldberg. 2000. The contribution of argument structure constructions to sentence meaning. Journal of Memory and Language, 43(4):640–651. ↗
  4. Charles J. Fillmore, Paul Kay, and Mary C. O’Connor. 1988. Regularity and idiomaticity in grammatical constructions: The case of let alone. Language, 64(3):501–538. ↗
  5. Leonard Bloomfield. 1933. Language. Holt, Rinehart & Winston, New York, NY. find
  6. Adele Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press, Chicago, IL. find
  7. Noam Chomsky. 1988. Generative grammar. Studies in English linguistics and literature. find
  8. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136, Melbourne, Australia. Association for Computational Linguistics. ↗
  9. Peter W Culicover and Ray Jackendoff. 1999. The view from the periphery: The english comparative correlative. Linguistic inquiry, 30(4):543–571. ↗
  10. Dorottya Demszky, Devyani Sharma, Jonathan H. Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2021. find
  11. Marcel Den Dikken. 2005. Comparative correlatives comparatively. Linguistic Inquiry, 36(4):497–532. ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017. Automatically tagging constructions of causation and their slot-fillers. Transactions of the Association for Computational Linguistics, 5:117–133. ↗
  14. Jonathan Dunn. 2019. Frequency vs. association for constraint selection in usage-based construction grammar. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 117–128, Minneapolis, Minnesota. Association for Computational Linguistics. find
  15. Charles J Fillmore. 1986. Varieties of conditional sentences. In Eastern States Conference on Linguistics, volume 3, pages 163–182. find
  16. Charles J. Fillmore. 1989. Grammatical construction: Theory and the familiar dichotomies. In Rainer Dietrich and Carl F. Graumann, editors, Language processing in social context, pages 17–38. North-Holland, Amsterdam. ↗
  17. Adele Goldberg. 2006. Constructions at work: The nature of generalization in language. Oxford University Press, Oxford, UK. find
  18. Adele E Goldberg. 2003. Constructions: A new theoretical approach to language. Trends in cognitive sciences, 7(5):219–224. ↗
  19. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. arXiv preprint arXiv:1901.05287. ↗
  20. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. ↗
  21. Martin Hilpert. 2006. A synchronic perspective on the grammaticalization of Swedish future constructions. Nordic Journal of Linguistics, 29(2):151–173. ↗
  22. Thomas Hoffmann, Jakob Horsch, and Thomas Brunner. 2019. The more data, the better: A usage-based account of the english comparative correlative construction. Cognitive Linguistics, 30(1):1–36. ↗
  23. Thomas Hoffmann and Graeme Trousdale. 2013. The Oxford handbook of construction grammar. Oxford University Press. find
  24. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  25. Matthew Honnibal and Ines Montani. 2018. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  26. Eiichi Iwasaki and Andrew Radford. 2009. Comparative correlatives in english: A minimalist-cartographic analysis. find
  27. Matt A Johnson and Adele E Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes, 28(10):1439–1452. ↗
  28. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.
  29. Paul Kay and Charles J. Fillmore. 1999. Grammatical constructions and linguistic generalizations: The What’s X doing Y? construction. Language, 75(1):1– 33. ↗
  30. George Lakoff. 1987. Women, fire, and dangerous things: What categories reveal about the mind. University of Chicago Press, Chicago, IL. find
  31. Ronald W. Langacker. 1987. Foundations of cognitive grammar: Theoretical prerequisites. Stanford University Press, Stanford, CA. find
  32. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computational Linguistics. find
  33. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  34. Tânia Marques and Katrien Beuls. 2016. Evaluation strategies for computational construction grammars. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1137–1146, Osaka, Japan.
  35. The COLING 2016 Organizing Committee. find
  36. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. ↗
  37. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗
  38. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics. ↗
  39. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  40. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT) 2018.
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. find
  42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67. ↗
  43. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguistics. ↗
  44. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxGBERT: BERT meets construction grammar. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4020–4032, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  45. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations (ICLR) 7. ↗
  46. Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, and Shu-Kai Hsieh. 2022. CxLM: A construction and context-aware language model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6361– 6369, Marseille, France. European Language Resources Association. find
  47. Ivan Vulić, Edoardo M. Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pre-trained language models for lexical semantics. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  48. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. ↗
  49. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  51. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. Algorithm 1 Context-Free Grammar for Artificial Data Creation Training Set S → SPOS | SNEG SPOS → POS1 PUNCT POS2 ’.’ | POS1 INSERT PUNCT POS2 ’.’ SNEG → NEG1 PUNCT NEG2 ’.’ | NEG1 INSERT PUNCT NEG2 ’.’ PUNCT → ’,’ | ’;’ | ” CORE_POS → ADV_I ’the’ NUM NOUN VERB CORE_NEG → ADV_I NUM VERB ’the’ NOUN POS_UPPER → ’0 The’ CORE_POS POS_LOWER → ’0 the’ CORE_POS NEG_UPPER → ’0 The’ CORE_NEG NEG_LOWER → ’0 the’ CORE_NEG POS1 → POS_UPPER | POS_UPPER ADD | START POS_LOWER | START POS_LOWER ADD POS2 → POS_LOWER | POS_LOWER ADD NEG1 → NEG_UPPER | NEG_UPPER ADD | START NEG_LOWER | START NEG_LOWER ADD NEG2 → NEG_LOWER | NEG_LOWER ADD INSERT → INSERT1 | INSERT2 INSERT2 → ADDITION BETWEEN_ADD_AND_SENT SENT PRON → ’we’ | ’they’ ADDITION → ’, and by the way ,’ | ’, and I want to add that’ | ’, and’ PRON ’just want to say that’ | ’, and then’ PRON ’said that’ | ’, and then’ PRON ’said that’ SAY → ’say’ | ’think’ | ’mean’ | ’believe’ BETWEEN_ADD_AND_SENT → PRON SAY ’that’ | PRON SAY ’that’ | PRON SAY ’that’ | PRON SAY ’that’ LOC_SENT → PRON ’said this in’ LOC ’too’ LOC → CITY ’and’ LOC | CITY CITY → ’Munich’ | ’Washington’ | ’Cologne’ | ’Prague’ | ’Istanbul’ SENT → ’this also holds in other cases’ | ’this is not always true’ | ’this is always true’ | ’this has only recently been the case’ | ’this has not always been the case’ | ’this has always been the case’ INSERT1 → ’without stopping’ | ’without a break’ | ’without a pause’ | ’uninterrupted’ | START → ’Nowadays ,’ | ’Nowadays’ | ’Therefore ,’ | ’Therefore’ | ’We can’ CANWORD ’that’ | ’It is’ KNOWNWORD ’that’ | ’It follows that’ | ’Sometimes’ | ’Sometimes ,’ | ’It was recently announced that’ | ’People have told me that’ | ’I recently read in a really interesting book that’ | ’I have recently read in an established , well-known newspaper that’ | ’It was reported in a special segment on TV today that’ CANWORD → ’say’ | ’surmise’ | ’accept’ | ’state’ KNOWNWORD → ’clear’ | ’known’ | ’accepted’ | ’obvious’ ADD → TEMP | UNDER1 | TEMP UNDER1 | UNDER1 TEMP ADV_I → ADV | ADV ’and’ ADV TEMP → TEMP1 TEMP2 TEMP1 → ’before’ | ’after’ | ’during’ TEMP2 → ’the morning’ | ’the afternoon’ | ’the night’ UNDER1 → ’under the’ UNDER2 UNDER2 → ’bed’ | ’roof’ | ’sun’ VERB → ’push’ | ’attack’ | ’chase’ | ’beat’ | ’believe’ | ’boil’ | ’box’ | ’burn’ | ’call’ | ’date’ NOUN → ’lions’ | ’pandas’ | ’camels’ | ’pigs’ | ’horses’ | ’sheep’ | ’chickens’ | ’foxes’ | ’cows’ | ’deer’ ADV → ’worse’ | ’earlier’ | ’slower’ | ’deeper’ | ’bigger’ | ’smaller’ | ’flatter’ | ’weaker’ | ’stronger’ | ’louder’ NUM → ’twelve’ | ’thirteen’ | ’fourteen’ | ’fifteen’ | ’sixteen’ | ’seventeen’ | ’eighteen’ | ’nineteen’ | ’twenty’ | ’twenty-one’ Algorithm 2 Context-Free Grammar for Artificial Data Creation Test Set S → SPOS | SNEG SPOS → POS1 PUNCT POS2 ’.’ | POS1 INSERT PUNCT POS2 ’.’ SNEG → NEG1 PUNCT NEG2 ’.’ | NEG1 INSERT PUNCT NEG2 ’.’ PUNCT → ’,’ | ’;’ | ” CORE_POS → ADV_I ’the’ NUM NOUN VERB CORE_NEG → ADV_I NUM VERB ’the’ NOUN POS_UPPER → ’0 The’ CORE_POS POS_LOWER → ’0 the’ CORE_POS NEG_UPPER → ’0 The’ CORE_NEG NEG_LOWER → ’0 the’ CORE_NEG POS1 → POS_UPPER | POS_UPPER ADD | START POS_LOWER | START POS_LOWER ADD POS2 → POS_LOWER | POS_LOWER ADD NEG1 → NEG_UPPER | NEG_UPPER ADD | START NEG_LOWER | START NEG_LOWER ADD NEG2 → NEG_LOWER | NEG_LOWER ADD INSERT → INSERT1 | INSERT2 INSERT2 → ADDITION BETWEEN_ADD_AND_SENT SENT PRON → ’I’ | ’you’ ADDITION → ’, and by the way ,’ | ’, and I want to add that’ | ’, and’ PRON ’just want to say that’ | ’, and then’ PRON ’said that’ | ’, and then’ PRON ’said that’ SAY → ’say’ | ’think’ | ’mean’ | ’believe’ BETWEEN_ADD_AND_SENT → PRON SAY ’that’ | PRON SAY ’that’ | PRON SAY ’that’ | PRON SAY ’that’ LOC_SENT → PRON ’said this in’ LOC ’too’ LOC → CITY ’and’ LOC | CITY CITY → ’London’ | ’New York’ | ’Berlin’ | ’Madrid’ | ’Paris’ SENT → ’this also holds in other cases’ | ’this is not always true’ | ’this is always true’ | ’this has only recently been the case’ | ’this has not always been the case’ | ’this has always been the case’ INSERT1 → ’without stopping’ | ’without a break’ | ’without a pause’ | ’uninterrupted’ | START → ’Nowadays ,’ | ’Nowadays’ | ’Therefore ,’ | ’Therefore’ | ’We can’ CANWORD ’that’ | ’It is’ KNOWNWORD ’that’ | ’It follows that’ | ’Sometimes’ | ’Sometimes ,’ | ’It was recently announced that’ | ’People have told me that’ | ’I recently read in a really interesting book that’ | ’I have recently read in an established , well-known newspaper that’ | ’It was reported in a special segment on TV today that’ CANWORD → ’say’ | ’surmise’ KNOWNWORD → ’clear’ | ’known’ ADD → TEMP | UNDER1 | TEMP UNDER1 | UNDER1 TEMP ADV_I → ADV | ADV ’and’ ADV TEMP → TEMP1 TEMP2 TEMP1 → ’before’ | ’after’ | ’during’ TEMP2 → ’the day’ | ’the night’ | ’the evening’ UNDER1 → ’under the’ UNDER2 UNDER2 → ’bridge’ | ’stairs’ | ’tree’ VERB → ’slam’ | ’break’ | ’bleed’ | ’shake’ | ’smash’ | ’throw’ | ’strike’ | ’shoot’ | ’swallow’ | ’choke’ NOUN → ’cats’ | ’dogs’ | ’girls’ | ’boys’ | ’men’ | ’women’ | ’people’ | ’humans’ | ’mice’ | ’alligators’ ADV → ’faster’ | ’quicker’ | ’harder’ | ’higher’ | ’later’ | ’longer’ | ’shorter’ | ’lower’ | ’wider’ | ’better’ NUM → ’two’ | ’three’ | ’four’ | ’five’ | ’six’ | ’seven’ | ’eight’ | ’nine’ | ’ten’ | ’eleven’ *Figure 2: Full results for BERT LARGE on artificial data. Columns indicate the variable that the training and test set controls for.* *Figure 3: Full results for RoBERTa LARGE on artificial data. Columns indicate the variable that the training and test set controls for.* *Figure 4: Full results for DeBERTa LARGE on artificial data. Columns indicate the variable that the training and test set controls for.* *Figure 5: Full results for BERT LARGE on corpus data. Columns indicate the variable that the training and test set controls for.* *Figure 6: Full results for RoBERTa LARGE on corpus data. Columns indicate the variable that the training and test set controls for.* *Figure 7: Full results for DeBERTa LARGE on corpus data. Columns indicate the variable that the training and test set controls for.* Model Test Scenario - S5 S6 S7 BERT B S1 S2 S3 37.65% 64.64% 38.04% 37.62% 62.79% 44.78% 44.39% 56.66% 44.09% 47.9% 55.41% 48.29% BERT L S1 S2 S3 36.85% 67.13% 36.46% 31.91% 73.48% 43.43% 47.21% 54.39% 47.79% 44.03% 64.45% 44.36% RoBERTa B S1 S2 S3 61.6% 52.85% 62.21% 58.76% 51.35% 55.17% 42.13% 71.33% 43.04% 62.32% 60.25% 62.76% RoBERTa L S1 S2 S3 55.72% 68.01% 55.36% 58.37% 74.53% 52.02% 65.08% 62.73% 65.28% 69.53% 77.76% 69.23% DeBERTa B S1 S2 S3 49.72% 49.81% 50.28% 49.72% 48.67% 50.19% 49.86% 49.7% 49.97% 49.2% 49.06% 50.0% DeBERTa L S1 S2 S3 50.88% 51.41% 50.58% 49.86% 48.09% 49.94% 50.03% 47.21% 50.41% 49.39% 48.04% 49.42% DeBERTa XL S1 S2 S3 47.73% 49.34% 47.9% 45.08% 46.27% 49.14% 43.31% 45.58% 42.68% 43.67% 41.74% 45.58% DeBERTa XXL S1 S2 S3 47.35% 48.73% 47.57% 53.56% 52.85% 49.36% 54.92% 54.03% 55.25% 54.12% 53.81% 53.59% ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weissweiler-2022-better-your-syntax-better-your/