✂️ Tokenization & segmentation
Cutting text into words and pieces: tokenizers, subwords, word and sentence segmentation.
- Adaptive Sentence Boundary DisambiguationDavid D. Palmer, Marti A. Hearst1994 · 91 citations
- WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelsBenjamin Minixhofer, Fabian Paischer, Navid Rekabsaz2022 · 90 citations
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPSabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, et al.2021 · 89 citations
- PySBD: Pragmatic Sentence Boundary DisambiguationNipun Sadvilkar, Mark Neumann2020 · 89 citations
- Sentence Boundary Detection: A Long Solved Problem?Jonathon Read, Rebecca Dridan, Stephan Oepen, et al.2012 · 81 citations
- Tokenization Is More Than CompressionCraig W Schmidt, Varshini Reddy, Haoran Zhang, et al.2024 · 66 citations
- Evaluating Various Tokenizers for Arabic Text ClassificationZaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb, et al.2021 · 66 citations
- Experiments on Sentence Boundary DetectionMark Stevenson, Robert Gaizauskas2000 · 63 citations
- Tokenization counts: the impact of tokenization on arithmetic in frontier LLMsAaditya K. Singh, DJ Strouse2024 · 53 citations
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language ModelsXin Zhang, Dong Zhang, Shimin Li, et al.2023 · 51 citations
- Getting the most out of your tokenizer for pre-training and domain adaptationGautier Dagan, Gabriel Synnaeve, Baptiste Roziere2024 · 44 citations
- Tokenization: Returning to a Long Solved Problem – A Survey, Contrastive Experiment, Recommendations, and ToolkitRebecca Dridan, Stephan Oepen2012 · 43 citations
- Word Segmentation, Unknown-word Resolution, and Morphological Agreement in a Hebrew Parsing SystemYoav Goldberg, Michael Elhadad2013 · 39 citations
- FAST: Efficient Action Tokenization for Vision-Language-Action ModelsKarl Pertsch, Kyle Stachowicz, Brian Ichter, et al.2025 · 38 citations
- A unified approach to sentence segmentation of punctuated text in many languagesRachel Wicks, Matt Post2021 · 38 citations
- Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMsNicolas Boizard, Kevin El Haddad, Celine Hudelot, et al.2024 · 34 citations
- On the Proper Treatment of Tokenization in PsycholinguisticsMario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, et al.2024 · 23 citations
- Simultaneous Tokenization and Part-Of-Speech Tagging for Arabic without a Morphological AnalyzerSeth Kulick2010 · 17 citations
- Optimizing the Size of Subword Vocabularies in Dialect ClassificationVani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, et al.2023 · 11 citations
- On the Importance of Tokenization in Arabic Embedding ModelsMohamed Alkaoud, Mairaj Syed2020 · 11 citations
- A Cross-Task Flexible Transition Model for Arabic Tokenization, Affix Detection, Affix Labeling, POS Tagging, and Dependency ParsingStephen Tratz2013 · 7 citations
- Sentence Boundary Detection on Line Breaks in JapaneseYuta Hayashibe, Kensuke Mitsuzawa2020 · 6 citations
- Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language ModelsMohamed Taher Alrefaie, Nour Eldin Morsy, Nada Samir2024 · 5 citations
- The Effect of Automatic Tokenization, Vocalization, Stemming, and POS Tagging on Arabic Dependency ParsingEmad Mohamed2011 · 5 citations
- MoVoC: Morphology-Aware Subword Construction for Ge'ez Script LanguagesHailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl2025 · 4 citations
- MoVoC: Morphology-Aware Subword Construction for Geez Script LanguagesHailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl2025 · 4 citations
- Word Segmentation for Akkadian CuneiformTimo Homburg, Christian Chiarcos2016 · 4 citations
- Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM BenchmarksGo Inoue, Bashar Alhafni, Nizar Habash, et al.2026 · 3 citations
- Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and HebrewGiuseppe Samo, Paola Merlo2026 · 2 citations
- Arabic Sentence Segmentation Across Genres and Punctuation ConditionsMohammed Elkholy, Khalid N. Elmadani, Nizar Habash, et al.2026 · 1 citation
- Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMsYara Alakeel, Chatrine Qwaider, Hanan Aldarmaki, et al.2026 · 1 citation
- Morphemes without Borders: Evaluating Root–Pattern Morphology in Arabic Tokenizers and LLMsYara Yousif Alakeel, Chatrine Qwaider, Hanan Aldarmaki, et al.2026 · 1 citation
- Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for MalteseKurt Abela, Kurt Micallef, Marc Tanti, et al.2024 · 1 citation
- A Subword Embedding Approach for Variation Detection in Luxembourgish User CommentsAnne-Marie Lutgen, Alistair Plum, Christoph Purschke2026
- Bringing Mapudungun into the Modern MT Ecosystem: Morphology-Aware Tokenization for NLLB-200 Fine-TuningIsaac Thompson, Brandon Rogers, Eric Ringger2026
- LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language AdaptationHailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl2026
- SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language ModelsDong Liu, Yanxuan Yu2026
- The Impact of Tokenization Algorithms on Hungarian Language Model PerformanceMátyás Osváth, Máté Norbert Molnár, Roland Gunics, et al.2026
- Toxic Subword Pruning for Dialogue Response Generation on Large Language ModelsHongyuan Adam Lu, Wai Lam2026
- Word predictability estimates from language models are not robust to tokenizer vocabularyKien Nguyen, Suhas Arehalli2026
- AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3Mark Kashirskiy, Artiom Lipinski, Ilya Makarov2025
- Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMTBrendan T. Hatch, Stephen D. Richardson2025
This page in Markdown → https://md.research.elamur.ai/free/en/topics/tokenization/page/2/