Elamur Research⌕
☰
home / papers / 2026 / Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Paper · 2026 · Atsuki Yamaguchi, Maggie Mi and Nikolaos Aletras

Summary

Next-token prediction on raw text gives language models broad knowledge, but it does not directly train linguistic competence. Models can copy surface patterns without learning the underlying grammatical structure. The authors propose L2T, a pre-training framework that adds language learning tasks to standard CLM. Inspired by how humans acquire language, L2T turns raw text into structured input-output pairs, with no external annotation needed. Example tasks include fixing typos, restoring shuffled word order, and filling in missing passages. The authors trained Qwen2.5-based models with 500M and 1B parameters from scratch on FineWeb-Edu, using a fixed budget of 100B tokens. They compared two data regimes: one where source text is plentiful and one where it is limited. Mixing L2T data with raw text raised BLiMP scores in every configuration and sped up the acquisition of grammatical knowledge. Performance on reasoning and comprehension benchmarks stayed broadly competitive. The work suggests that the variety of training signals, not only the amount of data, shapes grammatical ability. This is especially relevant when unique text is scarce.

Key points

  • The tasks. L2T defines 14 tasks across four granularity levels: character (Char), word, sentence (Sent) and discourse (Disc). Examples are Typo, Shuffle, Reordering and Fill Middle. Each training sample is assigned one task at random.
  • The two data regimes. Both draw on FineWeb-Edu.
    • Disjoint: roughly 36B raw tokens plus 64B L2T tokens.
    • Shared: the same 42B source tokens are reused both as raw text and for generating L2T data.
    • The models use the Mistral 32K tokenizer in place of Qwen2.5's 130K vocabulary.
  • Overall BLiMP results.
    • Disjoint: 500M rose from 78.6 to 80.2, and 1B from 79.0 to 80.8.
    • Shared: gains were larger, at 2.8 points (500M) and 2.3 points (1B).
  • Results by phenomenon.
    • Island effects improved most, by 6.9 to 11.3 points.
    • Determiner-noun agreement (DN Agr) and ellipsis (Ellips) did not improve.
    • L2T models led the baselines from as early as 5B training tokens.
  • Limitations. Each configuration was trained only once (a single seed). The Shared 1B model lost some reasoning performance. Per-task analyses were run at a reduced 25B-token scale. Code and data pipeline: github.com/gucci-j/l2t.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Atsuki Yamaguchi, Maggie Mi and Nikolaos Aletras. 2026. Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.27

@inproceedings{yamaguchi2026enhancing,
  title = {{Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks}},
  author = {Atsuki Yamaguchi, Maggie Mi and Nikolaos Aletras},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.27},
  url = {https://doi.org/10.18653/v1/2026.acl-short.27}
}

References

88 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nameera Akhtar. 1999. Acquiring basic word order: evidence for data-driven learning of syntactic structure. Journal of Child Language, 26(2):339–356. find
  2. Ahmed Alajrami and Nikolaos Aletras. 2022. How does the pre-training objective affect what large language models learn about linguistic properties? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131–147, Dublin, Ireland. Ass… find
  3. Ahmed Alajrami, Katerina Margatina, and Nikolaos Aletras. 2023. Understanding the role of input token characters in language models: How does information loss affect performance? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9085–9108, Singapore. As… find
  4. Afra Alishahi. 2011. Computational Modeling of Human Language Acquisition, first edition. Synthesis Lectures on Human Language Technologies. Springer Cham. find
  5. Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, and 30 others. 2024… — PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  6. Stéphane Aroca-Ouellette and Frank Rudzicz. 2020. On Losses for Modern Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4970–4981, Online. Association for Computational Linguistics. find
  7. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint, arXiv:2207.14255.
  8. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  10. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
  11. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  12. Roger Brown and Jean Berko. 1960. Word association and the acquisition of grammar. Child Development, 31(1):1–14. find
  13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  14. Susanne Carroll, Merrill Swain, and Yves Roberge. 1992. The role of feedback in adult second language acquisition: Error correction and morphological generalizations. Applied Psycholinguistics, 13(2):173–198. find
  15. Tyler A. Chang and Benjamin K. Bergen. 2024. Language model behavior: A comprehensive survey. Computational Linguistics, 50(1):293–350.
  16. Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. 2022. Improving in-context few-shot learning via self-supervised training. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lingui… find
  17. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024a. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550, Miami, Florida, USA. Associat…
  18. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024b. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. find
  19. Jiali Cheng and Hadi Amiri. 2025. Linguistic blind spots of large language models. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 1–17, Albuquerque, New Mexico, USA. Association for Computational Linguistics. find
  20. Cheng-Han Chiang and Hung-yi Lee. 2022. On the transferability of pre-trained language models: A study from artificial datasets. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10518–10525. find
  21. Noam Chomsky. 1965. Aspects of the Theory of Syntax. The MIT Press. find
  22. Noam Chomsky. 2002. Syntactic Structures. Gruyter Mouton, Berlin, New York. find
  23. De Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint, arXiv:1803.05457.
  24. Yiming Cui, Wanxiang Che, Shijin Wang, and Ting Liu. 2022. LERT: A linguistically-motivated pre-trained language model. arXiv preprint, arXiv:2211.05344. — [arXiv:2211.05344](https://arxiv.org/abs/2211.05344) find
  25. Jennifer Culbertson and David Adger. 2014. Language learners privilege structured meaning over surface frequency. Proceedings of the National Academy of Sciences, 111(16):5842–5847. find
  26. Jillian Da Costa and Rui Chaves. 2020. Assessing the ability of transformer-based neural models to represent structurally unbounded dependencies. In Proceedings of the Society for Computation in Linguistics 2020, pages 12–21, New York, New York. Association for Computational Linguistics. find
  27. Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Proceedings of the Twelfth International Conference on Learning Representations.
  28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  29. Luca Di Liello, Matteo Gabburo, and Alessandro Moschitti. 2022. Effective pretraining objectives for transformer-based autoencoders. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5533–5547, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  30. Yassir El Mesbahi, Atif Mahmud, Abbas Ghaddar, Mehdi Rezagholizadeh, Phillippe Langlais, and Prasanna Parthasarathi. 2023. On the utility of enhancing BERT syntactic bias with token reordering pretraining. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), page… find
  31. Jeffrey L. Elman. 1993. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71–99. find
  32. Evelina Fedorenko and Rosemary Varley. 2016. Language and thought are not the same thing: Evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369(1):132– 153. find
  33. Lukas Galke, Yoav Ram, and Limor Raviv. 2024. Deep neural networks and humans both benefit from compositional language structure. Nature communications, 15(10816):1–13. find
  34. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  35. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s…
  36. Yuxian Gu, Pei Ke, Xiaoyan Zhu, and Minlie Huang. 2022. Learning instructions with unlabeled data for zero-shot cross-task generalization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1617–1634, Abu Dhabi, United Arab Emirates. Association for Comp… find
  37. Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. 2024. Mitigating reversal curse in large language models via semantic-aware permutation training. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11453–11464, Bangkok, Thailand. Association for Com… find
  38. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  39. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics.
  40. Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. Between circuits and Chomsky: Pre-pretraining on formal languages imparts linguistic biases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9691–…
  41. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics.
  42. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  43. Gregory D. Keating. 2008. Task effectiveness and word learning in a second language: The involvement load hypothesis on trial. Language Teaching Research, 12(3):365–386. find
  44. M. G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
  45. Payal Khullar, Kushal Majmundar, and Manish Shrivastava. 2020. NoEl: An annotated corpus for noun ellipsis in English. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 34–43, Marseille, France. European Language Resources Association.
  46. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  47. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3472– 3484, Florence, Italy. Association for Comput… find
  48. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
  49. Tal Levy, Omer Goldman, and Reut Tsarfaty. 2023. Is probing all you need? indicator tasks as an alternative to probing embedding spaces. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5243–5254, Singapore. Association for Computational Linguistics. find
  50. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — Datasets: A Community Library for Natural Language Processing
  51. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. LogiQA: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3622–3628.
  52. Miguel López-Otal, Jorge Gracia, Jordi Bernad, Carlos Bobed, Lucía Pitarch-Ballesteros, and Emma Anglés-Herrero. 2025. Linguistic interpretability of transformer-based language models: a systematic review. arXiv preprint, arXiv:2504.08001. — [arXiv:2504.08001](https://arxiv.org/abs/2504.08001) find
  53. Simon McIntosh-Smith, Sadaf Alam, and Christopher Woods. 2025. Isambard-AI: a leadership-class supercomputer optimised specifically for artificial intelligence. In Proceedings of the Cray User Group, CUG ’24, page 44–54, New York, NY, USA. Association for Computing Machinery. find
  54. Maggie Mi. 2023. Mmi01 at the BabyLM challenge: Linguistically motivated curriculum learning for pre-training in low-resource settings. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 269–278, Singapore. Association for Computational Li… find
  55. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  56. Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre-training for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8318–… find
  57. Florence Myles, Janet Hooper, and Rosamond Mitchell. 1998. Rote or rule? exploring the role of formulaic language in classroom foreignlanguage learning. Language Learning, 48(3):323–364. find
  58. Miyu Oba, Akari Haga, Akiyo Fukatsu, and Yohei Oseki. 2023. BabyLM challenge: Curriculum learning based on sentence complexity approximating language acquisition. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 290–297, Singapore. Assoc…
  59. Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. 2022. Exploring the role of task transferability in large-scale multi-task learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find
  60. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association fo…
  61. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
  62. Amy Perfors, Joshua B. Tenenbaum, and Terry Regier. 2011. The learnability of abstract syntactic principles. Cognition, 118(3):306–338. find
  63. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  64. Kim Plunkett and Virginia Marchman. 1993. From rote learning to system building: acquiring verb morphology in children and connectionist nets. Cognition, 48(1):21–69. find
  65. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. arXiv preprint, ar… find
  66. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report. find
  67. Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowledge in language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7302– 7315, Dublin, Ireland. Association for Com… find
  68. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866.
  69. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, and 21 others… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  70. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  71. Raj Sanjay Shah, Khushi Bhardwaj, and Sashank Varma. 2024. Development of cognitive intelligence in pre-trained language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9632–9657, Miami, Florida, USA. Association for Computational Linguistics. find
  72. Elizabeth Spelke. 2022. What babies know: Core knowledge and composition. Oxford University Press. find
  73. Zia Tajeddin and Ali Rahimi. 2017. A conversation analysis of ellipsis and substitution in global business english textbooks. International Journal of Society, Culture &; Language, 5(1):1–14. find
  74. Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024. Holmes a benchmark to assess the linguistic competence of language models. Transactions of the Association for Computational Linguistics, 12:1616–1647. find
  75. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  76. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. find
  77. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  78. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In Proceedings of the Tenth International Conference on Learning Representations.
  79. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
  80. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  81. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  82. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint, arXiv:1810.12885.
  83. Shuai Zhang, Wang Lijie, Xinyan Xiao, and Hua Wu. 2022. Syntax-guided contrastive learning for pre-trained language model. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2430–2440, Dublin, Ireland. Association for Computational Linguistics. find
  84. Yue Zhang and Stephen Clark. 2015. Discriminative syntax-based word ordering for text generation. Computational Linguistics, 41(3):503–538.
  85. Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. 2021. Syntax-enhanced pre-trained model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference o… find
  86. Aditya Yadavalli, Alekhya Yadavalli, and Vera Tobin. 2023. SLABERT talk pretty one day: Modeling second language acquisition with BERT. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11763–11777, Toronto, Canada. Association… find
  87. Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, and Nikolaos Aletras. 2021. Frustratingly simple pretraining alternatives to masked language modeling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3116–3125, Online and Punta Cana, Dominic… find
  88. Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita, Gaku Morio, and Yasuhiro Sogawa. 2023. How does the task complexity of masked pretraining objectives affect downstream performance? In Findings of the Association for Computational Linguistics: ACL 2023, pages 10527–10537, Toronto, Canada. Associa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yamaguchi-2026-enhancing-linguistic-competence-language-models/