Elamur Research⌕
☰
home / papers / 2026 / LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation

Summary

Multilingual pretrained models such as XLM-R represent low-resource, morphologically rich languages poorly. Frequency-driven subword splitting (BPE, SentencePiece) breaks words in Ge'ez-script languages into meaningless fragments, and vocabulary-expansion methods usually give new tokens random vectors or averages of those fragments. The authors propose LGSE, which builds each new token's starting vector from its morphemes. A word is segmented morphologically, and pretrained subword or FastText morpheme vectors are averaged. When no usable segmentation exists, character n-gram vectors are used instead. During Language-Adaptive Pretraining, a regularization term keeps the new embeddings close to their initial values so they stay compatible with the original embedding space. To isolate the effect of initialization, the original vocabulary and tokenizer are kept and only the new embeddings are trained. For both Amharic and Tigrinya, LGSE beat the baseline initializations in each of the three evaluations: Text Classification, Named Entity Recognition, and Question Answering. The authors also release a human-annotated benchmark. The work shows that morphology-aware initialization is a practical way to improve representations for underserved languages.

Key points

  • Motivation: XLM-R uses a shared vocabulary of 250K subwords, but each language effectively gets only about 2.5K of them. BPE, for example, splits the Amharic word for "egg" (እንቁላል) into five single-character pieces rather than two meaningful units.
  • Method: New-token embeddings are built in three steps:
    • morpheme segmentation of the word;
    • averaging of pretrained subword or FastText morpheme vectors;
    • a character n-gram fallback when segmentation fails.
  • Training: Adaptation uses LAPT with a penalty on drift away from the initialized embedding values. Only the new embeddings are updated, and the original tokenizer is kept.
  • Resource: The authors present what they describe as the first human-annotated benchmark for Amharic and Tigrinya downstream tasks. It includes classification of high-quality educational content and is available via the project page (hailaykidu.github.io/LGSE-Project-).
  • Results: LGSE consistently beat the baseline methods on QA, NER and Text Classification in both languages. Exact scores were in the omitted middle part of the provided text, so they are not reproduced here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5i7acx7v348w

@inproceedings{teklehaymanot2026lgse,
  title = {{LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation}},
  author = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5i7acx7v348w},
  url = {https://doi.org/10.63317/5i7acx7v348w}
}

References

53 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Li… find our card
  2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904… find our card
  3. Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. 2025. Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies. arXiv preprint arXiv:2502.00894. — [arXiv:2502.00894](https://arxiv.org/abs/2502.00894) find
  4. Duygu Ataman and Marcello Federico. 2018. Compositional representation of morphologically-rich input for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 305–311, Melbourne, Australia. Association f… find
  5. Abhinand Balachandran. 2023. Tamil-llama: A new tamil language model based on llama 2. arXiv preprint arXiv:2311.05845. — [arXiv:2311.05845](https://arxiv.org/abs/2311.05845) find
  6. M Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Sercan Karakaş, Banu Diri, Savaş Yıldırım, and Demircan Çelik. 2025. Tokens with meaning: A hybrid tokenization approach for nlp. arXiv preprint arXiv:2508.14292. — [arXiv:2508.14292](https://arxiv.org/abs/2508.14292) find
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. find
  8. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. find
  9. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics. find
  10. ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, and KyungTae Lim. 2024a. Optimizing language augmentation for multilingual large language models: A case study on Korean. In P… find
  11. ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, et al. 2024b. Optimizing language augmentation for multilingual large language models: A case study on korean. arXiv preprint arXiv:2403.10882. find
  12. Kartikay Conneau, Alexis Workshop Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Assoc… find
  13. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177. find our card
  14. Wietse de Vries and Malvina Nissim. 2021. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846, Online. Association for Computational Linguistics. find
  15. Konstantin Dobler and Gerard de Melo. 2023. FOCUS: Effective embedding initialization for monolingual specialization of multilingual models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454, Singapore. Association for Computational Linguist… find
  16. C. M. Downey, Terra Blevins, Dhwani Serai, Dwija Parikh, and Shane Steinert-Threlkeld. 2024. Targeted multilingual adaptation for low-resource language families. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15647–15663, Miami, Florida, USA. Association for Computat… find
  17. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. find
  18. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. arXiv preprint arXiv:2404.17790. find our card
  19. Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (… find
  20. Fitsum Gaim, Wonsuk Yang, and Jong C. Park. 2021. Tlmd: Tigrinya language modeling dataset. [Dataset]. find
  21. Rob van der Goot, Anette Jensen, Emil Allerslev Schledermann, Mikkel Wildner Kildeberg, Nicolaj Larsen, Mike Zhang, and Elisa Bassignana. 2025. MorSeD: Morphological segmentation of Danish and its effect on language modeling. In Proceedings of the Joint 25th Nordic Conference on Computational Lingui… find
  22. Viktor Hangya, Silvia Severini, Radoslav Ralev, Alexander Fraser, and Hinrich Schütze. 2023. Multilingual word embeddings for low-resource languages using anchors and a chain of related languages. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 95–105, Singap… find
  23. Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic tokenization with a language model for neural text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620–1629. find
  24. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38… find
  25. Haris Jabbar. 2023. Morphpiece: A linguistic tokenizer for large language models. arXiv preprint arXiv:2307.07262. — [arXiv:2307.07262](https://arxiv.org/abs/2307.07262) find
  26. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049, Brussels, Belgium. Association for Computat… find
  27. Celio Larcher, Marcos Piau, Paulo Finardi, Pedro Gengo, Piero Esposito, and Vinicius Caridá. 2023. Cabrita: closing the gap for foreign languages. arXiv preprint arXiv:2308.11878. — [arXiv:2308.11878](https://arxiv.org/abs/2308.11878) find
  28. Tomasz Limisiewicz, Jiří Balhar, and David Mareček. 2023. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681, Toronto, Canada. Association for Compu… find
  29. Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André FT Martins, and Hinrich Schütze. 2024. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303. find
  30. Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. Bridging subword gaps in pretrain-finetune paradigm for natural language generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I… find
  31. Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067– 1097, Mexico City, Mexico.… find
  32. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find
  33. Md Mofijul Islam, Gustavo Aguilar, Pragaash Ponnusamy, Clint Solomon Mathialagan, Chengyuan Ma, and Chenlei Guo. 2022. A vocabulary-free multilingual neural tokenizer for end-to-end task learning. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 91–99, Dublin, Ireland. As… find
  34. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu… find our card
  35. Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. An empirical comparison of vocabulary expansion and initialization approaches for language models. In Proceedings of the 28th Conference on Computational Natural Language Le… find
  36. Arijit Nag, Soumen Chakrabarti, Animesh Mukherjee, and Niloy Ganguly. 2025. Efficient continual pre-training of LLMs for low-resource languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  37. Arijit Nag, Bidisha Samanta, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2023. Entropy-guided vocabulary augmentation of multilingual language models for low-resource tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8619–8629, Toronto, Canada. Associ… find
  38. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. SeaLLMs - large language models for Southeast Asia. In Proceedings… find
  39. Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various Korean NLP tasks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference o… find
  40. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in neural information processing systems, 36:36963–36990. find
  41. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673. find our card
  42. Trinh Pham, Khoi Le, and Anh Tuan Luu. 2024. UniBridge: A unified approach to cross-lingual transfer learning for low-resource languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3168–3184, Bangkok, Thailand. Associatio… find
  43. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find our card
  44. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… find
  45. Jimin Sun, Patrick Fernandes, Xinyi Wang, and Graham Neubig. 2023. A multi-dimensional evaluation of tokenizer-free multilingual pre-trained models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1725–1735, Dubrovnik, Croatia. Association for Computational Linguistics… find
  46. Tilahun Abedissa Taffa, Ricardo Usbeck, and Yaregal Assabie. 2024. Low resource question answering: An Amharic benchmarking dataset. In Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024, pages 124–132, Torino, Italia. ELRA and ICCL. find
  47. Hailay Kidu Teklehaymanot, Dren Fazlija, Niloy Ganguly, Gourab Kumar Patro, and Wolfgang Nejdl. 2024. TIGQA: An expert-annotated In question-answering dataset in Tigrinya. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL… find
  48. Hailay Kidu Teklehaymanot, Dren Fazlija, and Wolfgang Nejdl. 2025a. Movoc: Morphology-aware subword construction for ge’ez script languages. arXiv preprint arXiv:2509.08812. find our card
  49. Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam Gidey, and Wolfgang Nejdl. 2025b. Low-resource english–tigrinya mt: Leveraging multilingual models, custom tokenizers, and clean evaluation benchmarks. In 2025 3rd International Conference on Foundation and Large Language Models (FLLM), pages 121–1… find
  50. Hailay Kidu Teklehaymanot and Wolfgang Nejdl. 2025. Tokenization disparities as infrastructure bias: How subword systems create inequities in llm access and efficiency. In 2025 3rd International Conference on Foundation and Large Language Models (FLLM), pages 822–828. find
  51. Asahi Ushio, Yi Zhou, and Jose Camacho-Collados. 2023. Efficient multilingual language model compression through vocabulary trimming. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14725–14739, Singapore. Association for Computational Linguistics. find
  52. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019. Hai wang, dian yu, kai sun, janshu chen, and 791 dong yu. 2019. improving pre-trained multilingual 792 models with vocabulary expansion. arxiv preprint 793 arxiv:1909.12440. In Proceedings of the 23rd Conference on Computational Natural La… — [arXiv:1909.12440](https://arxiv.org/abs/1909.12440) find
  53. Hailemariam Mehari Yohannes and Toshiyuki Amagasa. 2022. A method of named entity recognition for tigrinya. ACM SIGAPP Applied Computing Review, 22(3):56–68. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/teklehaymanot-2026-lgse-lexically-grounded-subword-embedding/