Elamur Research⌕
☰
home / papers / 2026 / Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

Paper · 2026 · H S V N S Kowndinya Renduchintala and Sumit Bhatia

Summary

Language models score well on BLiMP overall, yet on certain paradigms they stay below chance even after massive pre-training, and adding more tokens does not reliably fix this. The authors ask whether these gaps come from architectural limits or from rare exposure to the relevant constructions in web text. They train GPT-2 Small (124M) on 100M FineWeb tokens. In intervention runs, they replace 1% of the corpus with synthetic documents generated by GPT-OSS 120B. Each document is a natural-sounding text in a randomly chosen genre and subgenre. It must contain several sampled frequent lemmas and embeds a sentence that follows the syntactic template of one target BLiMP paradigm. The targeted runs are compared with an unmodified baseline and with a control that uses random synthetic data. Targeted injection improves 8 of the 9 weakest paradigms, and aggregate BLiMP results stay the same or rise slightly. The findings suggest that, at human-like data budgets, what the training data contains can matter more than its volume. This is relevant to BabyLM-style research.

Key points

  • Training setup: GPT-2 Small 124M with FlashAttention-2 and a 1024-token context, trained on 100M tokens for 20 epochs. Intervention runs use 99M FineWeb tokens plus 1M synthetic tokens. Only about 3K of the synthetic sentences actually instantiate the target paradigm.
  • Largest gains:
    • only_npi_scope rose from 20.9% to 69.4% (+48.5).
    • principle_A_reconstruction rose from 37.2% to 78.9% (+41.7). This beats Llama-3 70B (49.6%), which was trained on more than 15T tokens.
  • Resistant case: principle_A_c_command dropped from 49.0% to 46.0%. It stayed below chance at every injection level tested.
  • Ablation over 0.01%, 0.1% and 1% injection:
    • Most paradigms improve steadily as the share grows. For example, coordinate_structure_constraint_complex_left_branch went 46.5 → 52.7 → 59.3 → 62.4.
    • For principle_A_reconstruction, 0.01% (about 10K tokens, roughly 30 examples) already reached 72.3%.
    • only_npi_scope and existential_there_quantifiers_2 behaved non-monotonically.
  • Limitations:
    • Only one architecture was tested.
    • Only 9 of the 67 BLiMP paradigms were targeted.
    • No injection shares above 1% were tried.
    • Effects on non-targeted paradigms can be positive or negative.
    • Code is available on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as H S V N S Kowndinya Renduchintala and Sumit Bhatia. 2026. Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.224

@inproceedings{renduchintala2026heterogeneity,
  title = {{Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?}},
  author = {H S V N S Kowndinya Renduchintala and Sumit Bhatia},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.224},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.224}
}

References

17 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  3. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, and 1 others. 2025. Smollm2: When smol goes big–data-centric training of a small language model. arXiv preprint ar…
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Michael Y Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. arXiv preprint arXiv:2412.… find
  6. Cameron Robert Jones, Ishika Rathi, Sydney Taylor, and Benjamin K Bergen. 2025. People cannot distinguish gpt-4 from a human in a turing test. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 1615–1639. find
  7. Shalom Lappin and Stuart M Shieber. 2007. Machine learning theory and practice as a source of insightinto universal grammar. Journal of Linguistics, 43(2):393– 427. find
  8. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2(3). find
  9. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929, Miami, Florida, USA. Association for Computational Linguisti…
  10. Peter Norvig. n-grams. https://norvig.com/ ngrams/. Accessed: 2026-01-06. ↗
  11. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.
  12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  13. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849.
  14. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  16. Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL Hansen, and Terrance D Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):24… find
  17. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/renduchintala-2026-heterogeneity-formal-linguistic-competence-language/