Elamur Research⌕
☰
home / papers / 2026 / Tokenization Cost, Retention, and Orthography Robustness for Ladin and Italian Varieties

Tokenization Cost, Retention, and Orthography Robustness for Ladin and Italian Varieties

Paper · 2026 · Alessio Staffini

Summary

The paper treats tokenizer mismatch as an early bottleneck for low-resource varieties. When text is split into too many subwords or bytes, it uses up context, raises truncation risk and can make models sensitive to small spelling changes. Staffini audits this on the ITDI benchmark of eleven Italian varieties, with Ladin (LLD) as the focus. The study compares four off-the-shelf tokenizers from popular multilingual models. It measures three groups of diagnostics:

  • cost: tokens per word, bytes per token, truncation pressure;
  • retention: the share of words broken into pieces, the share of tokens that continue a word, and how many word types stay intact;
  • fragmentation: token-length proxies.

It then applies controlled orthographic perturbations, such as stripping diacritics and normalizing apostrophes and dashes, and checks how lightweight variety-identification baselines degrade. Stripping diacritics causes clear macro-F1 losses, while apostrophe normalization has no measurable effect on this data. Subword tokenizers (WordPiece, SentencePiece) give a good balance of cost, retention and robustness. Byte-BPE loses slightly less accuracy but uses more tokens. Byte-level tokenization is much more expensive and more fragile in token-based classifiers. The value of the work is a reproducible, cheap check that practitioners can run before choosing a tokenizer for a low-resource variety.

Key points

  • Data: ITDI (VarDial 2022) after preprocessing has 215,997 train, 10,799 dev and 11,087 test sentences. Training data is capped at 20,000 sentences per variety. Dev holdouts of 1,000 sentences were created for LLD, EML, NAP and ROA_TARA, which lack an official dev split.
  • Tokenizers audited: google/byt5-small (256 vocab), gpt2 byte-BPE (50,257), bert-base-multilingual-cased WordPiece (119,547) and xlm-roberta-base SentencePiece (250,002).
  • Robustness: under strip_diacritics, macro-F1 drops range from about 0.036 to 0.111 across the reported configurations. Ladin-specific drops reach up to 0.172. apostrophe_normalize produced zero drop.
  • Association analysis (Spearman, token_tfidf baselines):
    • Absolute BPT is the only diagnostic whose bootstrap CI excludes zero (median ρ = −0.312).
    • ΔTPW shows the largest median correlation (ρ = −0.374), but its CI includes zero.
  • Limitations: baselines are TF-IDF linear models and a frozen encoder probe, not fine-tuned transformers. Code and derived statistics are released on GitHub (ladin-itdi-tokenizer-audit, v1.0.0).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Tokenization & segmentation · 119 works

Cite

Cite as Alessio Staffini. 2026. Tokenization Cost, Retention, and Orthography Robustness for Ladin and Italian Varieties. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.49

@inproceedings{staffini2026tokenization,
  title = {{Tokenization Cost, Retention, and Orthography Robustness for Ladin and Italian Varieties}},
  author = {Alessio Staffini},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.49},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.49}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Noëmi Aepli, Antonios Anastasopoulos, Adrian-Gabriel Chifu, William Domingues, Fahim Faisal, Mihaela Găman, Radu Tudor Ionescu, and Yves Scherrer. 2022. Findings of the VarDial evaluation campaign 2022. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages… find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2025. Glottolog 5.2. Accessed 2026-01-31. find
  5. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  6. Mir Tafseer Nayeem, Sawsan Alqahtani, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, and M Saiful Bari. 2025. Beyond fertility: Analyzing STRR as a metric for multilingual tokenization evaluation. Preprint, arXiv:2510.09947. find
  7. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find
  8. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  9. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. find
  10. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. find
  11. Appendix overview. This appendix expands on implementation details and analysis choices that support reproducibility and interpretation. We document marker handling (Appendix A), our word-extraction heuristic (Appendix B), baseline settings (Appendix D), and additional statistical results (Appendix… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/staffini-2026-tokenization-cost-retention-orthography-robustness/