Elamur Research⌕
☰
home / papers / 2026 / How Do Language Models Acquire Character-Level Information?

How Do Language Models Acquire Character-Level Information?

Paper · 2026 · Soma Sato and Ryohei Sasano

Summary

Language models work with subword tokens and never see individual letters directly, yet earlier probing studies found that their embeddings still reveal which characters a token contains. Sato and Sasano ask where this knowledge comes from. They pretrain small models (BERT-Tiny with WordPiece, nanoGPT with BPE) on the FineWeb 10B-token sample. They then vary the training conditions, changing the corpus or the tokenizer, and compare the results against models trained normally. Character knowledge is measured with an MLP probe that checks whether a static token embedding contains a given letter. Unlike earlier work, the probe balances token lengths between positive and negative examples, so length cannot serve as a shortcut. The authors sort the contributing factors into two groups. Factors tied to tokenization are merge rules and orthographic constraints. Factors independent of tokenization are semantic associations among words that share substrings, and syntactic cues carried by affixes. The study suggests LMs pick up spelling indirectly from several overlapping signals rather than treating tokens as opaque units. This helps explain the spelling and sound-based abilities LMs sometimes show.

Key points

  • Length-matched probing lowers accuracy. With matched lengths, scores are GPT-J 84.0, GPT-2 75.5, BERT-base-uncased 67.1, nanoGPT 66.7 and BERT-Tiny 60.8. Without matching, the same models score 88.0, 79.4, 69.1, 69.6 and 63.3.
  • Short tokens are easier to probe. Accuracy falls from 80.4% for tokens of up to 3 characters to 62.3% for tokens of 7 or more characters.
  • Bigram-based grouping hurts the probe. Grouping words by the letter before or after the target lowers accuracy to 57.3% and 57.8%, compared with a 61.2% baseline.
  • Removing affixes also hurts. Lemmatization (NLTK WordNetLemmatizer) costs 3.0 points and Porter stemming costs 3.9 points, so both inflectional and derivational morphemes help.
  • Limitations: the experiments cover English only, use FineWeb as the sole corpus and rely on small models, and possible explicit lexical content in the data was not fully controlled.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nagoya UniversityNagoya · Japan8 works · 8 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Soma Sato and Ryohei Sasano. 2026. How Do Language Models Acquire Character-Level Information?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.282

@inproceedings{sato2026how,
  title = {{How Do Language Models Acquire Character-Level Information?}},
  author = {Soma Sato and Ryohei Sasano},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.282},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.282}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marie Cassar and Rebecca Treiman. 1997. The beginnings of orthographic knowledge: Children’s knowledge of double letters in words. Journal of Educational Psychology, pages 631–644. find
  2. Cristiano Ciaccio, Marta Sartor, Alessio Miaschi, and Felice Dell’Orletta. 2025. Beyond the spelling miracle: Investigating substring awareness in character-blind language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pages 11361–11372… find
  3. A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–38. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (AC…
  5. Lukas Edman, Helmut Schmid, and Alexander Fraser. 2024. CUTE: Measuring llms’ understanding of their tokens. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 3017–3026. find
  6. Heather Hayes, Rebecca Treiman, and Brett Kessler. 2006. Children use vowels to help them spell consonants. Journal of Experimental Child Psychology, pages 27–42. find
  7. Tatsuya Hiraoka and Naoaki Okazaki. 2024. Knowledge of pretrained language models on surface information of tokens. arXiv:2402.09808. — [arXiv:2402.09808](https://arxiv.org/abs/2402.09808) find
  8. Itay Itzhak and Omer Levy. 2022. Models in a spelling bee: Language models implicitly learn the character composition of tokens. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022), pages 5061–5068. find
  9. Andrej Karpathy. 2022. nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts. https://github.com/karpathy/nanoGPT. ↗
  10. Ayush Kaushal and Kyle Mahowald. 2022. What do tokens know about their characters and how do they know it? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2022), pages 2487–2507. find
  11. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Volume 1: Long Papers, pages 66–75.
  12. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2018), pages 66–71.
  13. George A. Miller. 1992. WordNet: A lexical database for english. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman.
  14. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. arXiv:2406.17557.
  15. Martin F. Porter. 1980. An algorithm for suffix stripping. Program-electronic Library and Information Systems. find
  16. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
  17. Vilayanur S. Ramachandran and Edward M. Hubbard. 2001. Synaesthesia—A window into perception, thought and language. Nature Reviews Neuroscience, pages 43–53. find
  18. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2012), pages 5149–5152. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pages 1715–1725.
  20. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), pages 4593–4601.
  21. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. In arXiv preprint arXiv:1908.08962.
  22. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  23. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, and 12 others. 2016. Google’s n… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sato-2026-how-do-language-models-acquire/