Elamur Research⌕
☰
home / papers / 2026 / Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans

Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans

Summary

The paper asks whether character-level neural inflection models can act as cognitive models of how speakers learn morphological structure that has no functional basis. The test case is the Spanish L-shaped morphome, in which the 1sg present indicative (pongo) uses the same stem as the entire present subjunctive, while the remaining indicative cells use the regular stem. The authors train five encoder-decoder transformer variants on IPA-transcribed Spanish verb paradigms. The variants differ in two ways: sequential versus position-invariant positional encoding for morphosyntactic tags, and atomic versus decomposed tag representations. They also vary how common L-shaped verbs are in training. Models are then compared with human wug-test responses on nonce verbs.

Positional encoding turns out to matter most. Position-invariant models recover the correct paradigm grouping even when L-shaped verbs are rare, but sequential models capture it only partly. No model generalizes like humans, however. Models spread the special stem to subjunctive cells but not to 1sg.ind, which amounts to a mood-based rule. Speakers favour 1sg.ind instead. This points to a gap between reproducing statistical patterns and forming genuine morphological abstractions.

Key points

  • Data: 12-cell present-tense Spanish paradigms in IPA, taken from Kakolu Ramarao et al. (2025).
  • Training conditions: three L-shaped/NL-shaped ratios (10%, 50% and 90% L-shaped). The 10% setting is closest to the natural lexicon, where about 7% of verb types are L-shaped.
  • Splits and training runs:
    • 332 lemmas per condition, split 232/34/66 with no lemma overlap across splits.
    • Each input pairs two source form-tag pairs with one target tag. There are 660 possible triples per lemma, and 25% are subsampled per seed.
    • This yields 12 trained models per condition per architecture.
  • Human comparison: wug-test data from Nevins et al. (2015), covering 107 native speakers and 15 nonce verbs with fricative/stop stem alternations. Evaluation targets 1sg.ind, 2sg.sbjv and 3sg.sbjv, giving 120 test items per model run.
  • Availability: code and data are released at github.com/hhuslamlab/feature_informed_transformer_cmcl2026 (CMCL @ LREC 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Morphology

Institutions

🎓Ruhr University BochumBochum · Germany19 works · 1 147 citations🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany12 works · 846 citations🎓University of FloridaGainesville · United States7 works · 271 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ovkb8stpc2h

@inproceedings{ramarao2026character,
  title = {{Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans}},
  author = {Ramarao, Akhilesh Kakolu and Tang, Kevin and Baer-Henney, Dinah},
  booktitle = {Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2026},
  doi = {10.63317/3ovkb8stpc2h},
  url = {https://doi.org/10.63317/3ovkb8stpc2h}
}

References

42 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra https://credit.niso.org/ ↗
  2. Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  3. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30, Vancouver. Association for Computational Linguistics. find
  4. The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 1–21, Miami, FL, USA. Association for Computational Linguistics. find
  5. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske,… find
  6. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  7. The SIGMORPHON 2016 shared Task—Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics. find
  8. Cheonkam Jeong, Dominic Schmitz, Akhilesh Kakolu Ramarao, Anna Stein, and Kevin Tang. 2023. Linear discriminative learning: a competitive non-neural baseline for morphological inflection. In Proceedings of the 20th SIGMOR-PHON workshop on Computational Research in Phonetics, Phonology, and Morpholog… find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  10. Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022. Position information in transformers: An overview. Computational Linguistics, 48(3):733– 763. find
  11. Akhilesh Kakolu Ramarao, Kevin Tang, and Dinah Baer-Henney. 2025. Frequency matters: Modeling irregular morphological patterns in Spanish with transformers. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4474–4489, Vienna, Austria. Association for Computational Linguis… find
  12. Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma overlap artificially inflates models’ performance. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-2… find
  13. Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017. Neural multi-source morphological reinflection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 514–524… find
  14. Heidi Harley and Elizabeth Ritter. 2002. Person and number in pronouns: A feature-geometric analysis. Language, 78(3):482–526. find
  15. Katharina Kann and Hinrich Schütze. 2016. Single-model encoder-decoder with explicit morphological representation for reinflection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 555–560, Berlin, Germany. Association for Com… find
  16. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382–1390, Abu Dhabi, United Arab Emirates. Association for C… find
  17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980. find
  18. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second BabyLM challenge: Sample-efficient pre-training on developmentally plausible corpora. In Christo Kirov and Rya… find
  19. Jordan Kodner and Salam Khalifa. 2022. SIGMORPHON–UniMorph 2022 shared task 0: Modeling inflection in language acquisition. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 157–175, Seattle, Washington. Association for Computatio… find
  20. Ben Peters and André F. T. Martins. 2019. IT–IST at the SIGMORPHON 2019 shared task: Sparse two-headed models for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 50–56, Florence, Italy. Association for Computational Linguistics… find
  21. David E. Rumelhart, James L. MollClelland, and the PDP Research Group. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, volume 1. The MIT Press. find
  22. Jordan Kodner, Sarah Payne, Salam Khalifa, and Zoey Liu. 2023. Morphological inflection: A reality check. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6082– 6101, Toronto, Canada. Association for Computational Linguistics. find our card
  23. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New… find our card
  24. Ling Liu and Mans Hulden. 2022. Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 739–749, Dublin, Ireland. Association for Com… find
  25. Andrew Spencer and Mark Aronoff. 1994. Morphology by itself: Stems and inflectional classes. Language, 70:811. find
  26. Martin Maiden. 2011. Allomorphy, autonomous morphology and phonological conditioning in the history of the Daco-Romance present and subjunctive. Transactions of the Philological Society, 109(1):59–91. find
  27. Gregory T Stump. 2001. Inflectional morphology: A theory of paradigm structure, volume 93. Cambridge University Press. find
  28. Martin Maiden. 2018. The Romance verb: Morphomic structure and diachrony. Oxford University Press. find
  29. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. find our card
  30. Martin Maiden. 2021. The morphome. Annual Review of Linguistics, 7:89–108. find
  31. John Sylak-Glassman. 2016. The composition and use of the universal morphological feature schema (uni-morph schema). Technical report, Center for Language and Speech Processing, Johns Hopkins. find
  32. P H Matthews. 1972. Cambridge studies in linguistics: Inflectional morphology: A theoretical study based on aspects of Latin verb conjugation series number 6. Cambridge studies in linguistics. Cambridge University Press, Cambridge, England. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. find our card
  34. Kate McCurdy, Sharon Goldwater, and Adam Lopez. 2020. Inflecting when there’s no majority: Limitations of encoder-decoder neural networks as cognitive models for German plurals. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1745–1756, Online. Assoc… find
  35. Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tia… find
  36. SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 17th SIGMORPHON Andrew Nevins, Cilene Rodrigues, and Kevin Tang. 2015. The rise and fall of the L-shaped morphome: diachronic and experimental studies. Probus: International Journal of Latin and Roma… find
  37. Paul O’Neill. 2025. Defective verbs in portuguese: a morphomic approach. Cognitive Linguistics, 36(2):153–181. find
  38. Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1–39, Online. Association for Computational Linguistics. find
  39. Yu-An Wang and Yun-Nung Chen. 2020. What do position embeddings learn? an empirical study of pre-trained language model positional encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6840–6849, Online. Association for Computational Lingu… find
  40. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… find
  41. Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphol… find our card
  42. Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the transformer to character-level transduction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1901–1907, Online. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramarao-2026-character-aware-transformers-learn-irregular/