Elamur Research⌕
☰
home / papers / 2026 / On the Effect of Hyperparameters in Language Modeling for Computational Linguistics

On the Effect of Hyperparameters in Language Modeling for Computational Linguistics

Paper · 2026 · Ruoxi Ning, Yongpeng Zhu, Qingcheng Zeng et al.

Summary

Computational linguists often train language models from scratch to test linguistic hypotheses, but they usually rely on a single, loosely chosen configuration. This paper asks how much such findings depend on hyperparameters. The authors re-run three earlier studies while varying settings within realistic bounds: word acquisition (Chang and Bergen, 2022), syntactic generalization (Hu et al., 2020b) and word-order preferences (Kuribayashi et al., 2024).

Sensitivity grows with the abstraction level of the claim:

  • Lexical findings stay largely intact. These include staged word learning and the early fitting of unigram and bigram statistics.
  • Rankings of architectures by syntactic ability are moderately fragile. They can even flip, and they depend on how long models are trained.
  • Typological word-order conclusions, which rest on subtle statistical interactions, are the least stable.

The authors argue that some published results may reflect optimization artifacts rather than true inductive biases of model classes. They recommend several practices:

  • tuning models adequately,
  • reporting hyperparameter ranges, tuning procedures and other sources of randomness,
  • showing how conclusions change across the configuration space.

Key points

  • Experiment A replicates Chang and Bergen (2022) on BookCorpus plus WikiText-103.
    • It covers LSTM, bidirectional LSTM, BERT and GPT-2, using four extra configurations.
    • Each run took 2–14 days on four A40 GPUs.
    • Age-of-acquisition predictors changed in significance across configurations but never in sign.
    • Adjusted R² was 0.88–0.93 for models, versus 0.43 for children.
  • KL-divergence curves against unigram and bigram distributions first fall and then rise in every configuration. This confirms that models fit these statistics early in training.
  • Experiment B (SG scores) compares LSTM and GPT-2.
    • Hyperparameters such as hidden size, layer count, dropout and learning rate were significant predictors of SG scores.
    • Stopping training early, for example at 1/4 of the epochs, can falsely make the two architectures look similar.
  • Experiment C (word-order typology claims) was the most sensitive to hyperparameter sweeps.
  • Limitations:
    • Each phenomenon is tested on a single dataset or grammar.
    • Experiment A uses only one seed.
    • The search space is restricted; for example, optimizer type, LR schedule and weight decay were not varied.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🔬Vector InstituteToronto · Canada33 works · 4,535 citations🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ruoxi Ning, Yongpeng Zhu, Qingcheng Zeng, Tatsuki Kuribayashi and Freda Shi. 2026. On the Effect of Hyperparameters in Language Modeling for Computational Linguistics. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1939

@inproceedings{ning2026effect,
  title = {{On the Effect of Hyperparameters in Language Modeling for Computational Linguistics}},
  author = {Ruoxi Ning, Yongpeng Zhu, Qingcheng Zeng, Tatsuki Kuribayashi and Freda Shi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1939},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1939}
}

References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Morris Alper and Hadar Averbuch-Elor. 2023. Kiki or bouba? sound symbolism in vision-and-language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. find
  2. Christian Arnold, Luka Biedebach, Andreas Küpfer, and Marcel Neunhoeffer. 2024. The role of hyperparameters in machine learning models and how to tune them. Political Science Research and Methods, 12(4):841–848. find
  3. Christoph Aurnhammer and Stefan Frank. 2018. Comparing gated and simple recurrent neural network architectures as models of human sentence processing. find
  4. Lisa Beinborn and Nora Hollenstein. 2023. Cognitive plausibility in natural language processing. Springer.
  5. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 381–393. Associat… find
  6. Lukas Biewald. 2020. weights and biases. wandb.com. find
  7. Experiment tracking with Software available from Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag. find
  8. Gemma Boleda. 2020. Distributional semantics and linguistic theory. Annual Review of Linguistics, 6(1):213–234. find
  9. Xavier Bouthillier, César Laurent, and Pascal Vincent. 2019. Unreproducible research is reproducible. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 725–734.… find
  10. Tyler A. Chang and Benjamin K. Bergen. 2022. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics, 10:1–16. find
  11. Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. Bllip 1987-89 wsj corpus release 1. (No Title). find
  12. Dami Choi, Christopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, and George E. Dahl. 2019. On empirical comparisons of optimizers for deep learning. ArXiv preprint, abs/1910.05446. — [arXiv:1910.05446](https://arxiv.org/abs/1910.05446) find
  13. Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell, and Alex Warstadt. 2024. Investigating Critical Period Effects in Language Acquisition through Neural Language Models. find
  14. A. Feder Cooper, Yucheng Lu, Jessica Forde, and Christopher De Sa. 2021. Hyperparameter optimization is deceiving us, and how to stop it. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtu… find
  15. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… find
  16. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online (v2020.4). Zenodo. find
  17. Stephen Merity et al. 2016. preprint, abs/1609.07843.
  18. Wikitext-103. ArXiv Linnea Evanson, Yair Lakretz, and Jean Rémi King. 2023. Language acquisition: do children and language models follow similar learning stages? In Findings of the Association for Computational Linguistics: ACL 2023, pages 12205–12218. Association for Computational Linguistics.
  19. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… find
  20. Luca Franceschi, Michele Donini, Valerio Perrone, Aaron Klein, Cédric Archambeau, Matthias Seeger, Massimiliano Pontil, and Paolo Frasconi. 2024. Hyperparameter Optimization in Machine Learning. find
  21. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui…
  22. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. Association for Computat…
  23. Steven N. Goodman, Daniele Fanelli, and John PA Ioannidis. 2016. What does research reproducibility mean? Science translational medicine, 8(341):341ps12–341ps12. find
  24. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  25. Jennifer Hu, Sherry Yong Chen, and Roger Levy. 2020a. A closer look at the performance of neural language models on reflexive anaphor licensing. In Proceedings of the Society for Computation in Linguistics 2020, pages 323–333. Association for Computational Linguistics.
  26. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020b. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744. Association for Computational Linguis… find
  27. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646. Association for Computational Linguistics.
  28. Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke Van Der Plas. 2025. Evaluating morphological compositional generalization in large language models. In Proceedings of the 2025 Conference of the Nations of the Amer… find
  29. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. Association for Computati…
  30. Tatsuki Kuribayashi, Ryo Ueda, Ryo Yoshida, Yohei Oseki, Ted Briscoe, and Timothy Baldwin. 2024. Emergent Word Order Universals from Cognitively-Motivated Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14522–… find
  31. Hugo Larochelle, Dumitru Erhan, Aaron C. Courville, James Bergstra, and Yoshua Bengio. 2007. An empirical evaluation of deep architectures on problems with many factors of variation. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, J… find
  32. Alessandro Lenci. 2018. Distributional models of word meaning. Annual review of Linguistics, 4(1):151– 171. find
  33. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. Association for Computational Linguistics. find
  34. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963–980. Association for Computational Linguistics.
  35. James A. Michaelov, Roger P. Levy, and Benjamin K. Bergen. 2025. Language model behavioral phases are consistent across architecture, training data, and scale. find
  36. Raphaël Millière. 2024. Language models as models of language.
  37. Kanishka Misra and Kyle Mahowald. 2024. Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929. Association for Computational Linguistics.
  38. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science, 34(8):1388–1429. find
  39. Kevin P Murphy. 2012. Machine learning: a probabilistic perspective. MIT press. find
  40. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  41. Eva Portelance and Masoud Jasbi. 2024a. The roles of neural networks in language acquisition. Language and Linguistics Compass, 18(6):e70001. find
  42. Eva Portelance and Masoud Jasbi. 2024b. The Roles of Neural Networks in Language Acquisition. Language and Linguistics Compass, 18(6):e70001. find
  43. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  44. Terry Regier. 2005. The emergence of words: Attentional learning in form and meaning. Cognitive Science, 29(6):819–865. find
  45. Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press. find
  46. Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2020. On the role of supervision in unsupervised constituency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7611–7621. Association for Computational Linguistics. find
  47. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lak… find
  48. Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, and Barbara Plank. 2022. Experimental standards for deep learning in natural language processing research. In Findings of the Association for Computational Linguistics: EMNLP 2022,… find
  49. Alex Warstadt and Samuel R Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In Shalom Lappin and Jean-Philippe Bernardy, editors, Algebraic Structures in Natural Language, pages 17–60. CRC Press.
  50. Jennifer C. White and Ryan Cotterell. 2021. Examining the inductive bias of neural language models with artificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  51. Ethan Wilcox, Pranali Vani, and Roger Levy. 2021. A targeted assessment of incremental processing in neural language models and humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… find
  52. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior. find
  53. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santia…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ning-2026-effect-hyperparameters-language-modeling-computational/