Elamur Research⌕
☰
home / papers / 2021 / Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Summary

This is a survey paper by the BigScience Workshop tokenization working group. The authors examine which units of text NLP models should operate on: words, subwords, characters, bytes, or even pixels. The paper offers no new data and no method of its own. Instead, it connects research from the pre-neural and neural eras into a single narrative. First, the authors disentangle the concepts of typographic tokens, word forms, "pre-tokens", and subwords. They then discuss hybrid models that augment words with character-level information, as well as unsupervised discovery of word boundaries. Separate sections cover subword algorithms (BPE, WordPiece, Unigram LM), the competition among languages for a shared vocabulary in multilingual models, and the maximal decomposition of text into characters, bytes, or images. The main conclusion is that there is no universal solution, and most likely there never will be. Every approach has both strengths and weaknesses, and some requirements directly conflict with one another, for example robustness to noise versus computational efficiency that is fair across languages. Tokenization therefore continues to require deliberate decisions, even now that byte-level models are available.

Key points

  • The authors propose a taxonomy of segmentation algorithms along two dimensions: from linguistically motivated to learned from data, and from simple to complex. It includes the Moses tokenizer, Morfessor, Linguistica, BPE, WordPiece, Unigram LM, SentencePiece, as well as characters, bytes, and pixels.
  • A terminological shift: what used to be called "tokenization" is now called pre-tokenization, while tokens have become subwords, which need not carry any linguistic meaning. The correspondence between tokens and word forms is not one-to-one. Examples: don't = do + n't in the Penn Treebank, "multiword tokens" in UD.
  • Most tokenizers are irreversible: because of normalization and whitespace merging, the original text cannot be exactly recovered from their output.
  • Byte- and character-level models (CANINE, ByT5, Charformer) are more robust to noise and do not suffer from the OOV problem, but because of longer sequences they are slower to train and to run.
  • Visual models that render sentences as images (Salesky et al., 2021) are competitive in machine translation across different scripts and noticeably more robust to synthetic noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Tokenization & segmentation

Cite

Cite as Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot and Samson Tan. 2021. Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP. arXiv:2112.10508.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2112.10508 · arXiv 2112.10508

@misc{mielke2021between,
  title = {{Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP}},
  author = {Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot and Samson Tan},
  year = {2021},
  doi = {10.48550/arxiv.2112.10508},
  eprint = {2112.10508},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.10508}
}

References

198 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judit Ács. 2019. Exploring BERT’s vocabulary. find
  2. Gustavo Aguilar, Bryan McCann, Tong Niu, Nazneen Rajani, Nitish Keskar, and Thamar Solorio. 2021. Char2subword: Extending the subword embedding space using robust character compositionality. find
  3. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3159–3166. find
  4. David Aldón Mínguez, Marta Ruiz Costa-Jussà, and José Adrián Rodríguez Fonollosa. 2016. Neural machine translation using bitmap fonts. In Proceedings of the EAMT 2016 Fifth Workshop on Hybrid Approaches to Translation (Hy-Tra), pages 1–9. find
  5. Zaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2021. Evaluating various tokenizers for arabic text classification. ArXiv, abs/2106.07540. our card
  6. Chantal Amrhein and Rico Sennrich. 2021. How suitable are subword segmentation strategies for translating non-concatenative morphology? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 689– 705, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Duygu Ataman, Wilker Aziz, and Alexandra Birch. 2020. A latent morphology model for open-vocabulary neural machine translation. In International Conference on Learning Representations. find
  8. Duygu Ataman and Marcello Federico. 2018a. Compositional representation of morphologically-rich input for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 305–311, Melbourne, Australia. Association… find
  9. Duygu Ataman and Marcello Federico. 2018b. An evaluation of two vocabulary reduction methods for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 97–110. find
  10. Duygu Ataman, Orhan Firat, Mattia A. Di Gangi, Marcello Federico, and Alexandra Birch. 2019. On the importance of word boundaries in character-level neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 187–193, Hong Kong. Association for Computat… find
  11. R.H. Baayen, R Piepenbrock, and L Gulikers. 1995. CELEX2. In LDC96L14. Linguistic Data Consortium, University of Pennsylvania. find
  12. Tamali Banerjee and Pushpak Bhattacharyya. 2018. Meaningless yet meaningful: Morphology grounded subword-level nmt. In Proceedings of the second workshop on subword/character level models, pages 55–60. find
  13. Marco Baroni. 2000. Distributional Cues in Morpheme Discovery: A Computational Model and Empirical Evidence. Ph.D. thesis, University of California Los Angeles. find
  14. Sarah Beemer, Zak Boston, April Bukoski, Daniel Chen, Princess Dickens, Andrew Gerlach, Torin Hopkins, Parth Anand Jawale, Chris Koski, Akanksha Malhotra, Piyush Mishra, Saliha Muradoglu, Lan Sang, Tyler Short, Sagarika Shreevastava, Elizabeth Spaulding, Testumichi Umada, Beilei Xiang, Changbing Yan… find
  15. Kenneth R Beesley and Lauri Karttunen. 2003. Finite-state morphology: Xerox tools and techniques. CSLI, Stanford. find
  16. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In Advances in Neural Information Processing Systems, volume 13. MIT Press. our card
  17. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432. find
  18. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582–590, Los Angeles, Cal… find
  19. Toms Bergmanis and Sharon Goldwater. 2017. From segmentation to analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 337–346, Valencia, Spain… find
  20. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5(0):135–146. our card
  21. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. find
  22. Martin Braschler and Bärbel Ripplinger. 2004. How effective is stemming and decompounding for german text retrieval? Information Retrieval, 7(3):291–316. find
  23. Michael R Brent and Timothy A Cartwright. 1996. Distributional regularity and phonotactic constraints are useful for segmentation. Cognition, 61(1-2):93–125. find
  24. Michael R Brent, Sreerama K Murthy, and Andrew Lundberg. 1995. Discovering morphemic suffixes: a case study in MDL induction. In In Fifth International Workshop on AI and Statistics, Ft. Citeseer. find
  25. Samuel Broscheit. 2018. Learning distributional token representations from visual features. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 187–194, Melbourne, Australia. Association for Computational Linguistics. find
  26. Jacob Buckman and Graham Neubig. 2018. Neural lattice language models. Transactions of the Association for Computational Linguistics (TACL), 6. find
  27. Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104–2114, Online and Punta Cana, Dominican Republic. Association for Computational Lin… find
  28. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2017. Latent sequence decompositions. In International Conference on Learning Representations 2017. find
  29. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics. find
  30. Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394. find
  31. Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In EMNLP. find
  32. Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. 2019. Bridging the gap for tokenizer-free language models. find
  33. Grzegorz Chrupała. 2013. Text segmentation with character-level text embeddings. Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013; Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013 ; Conference date: 16-06-2013. find
  34. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546, Online. Association for Computational Linguist… find
  35. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations 2017. find
  36. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. arXiv preprint arXiv:1603.06147. our card
  37. Kenneth W. Church. 2000. Empirical estimates of adaptation: The chance of two Noriegas is closer to 𝑝/2 than 𝑝 2 . In Proceedings of the 18th Conference on Computational Linguistics - Volume 1, pages 180–186. Association for Computational Linguistics. find
  38. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. Canine: Pre-training an efficient tokenization-free encoder for language representation. arXiv preprint arXiv:2103.06874. our card
  39. Lionel Clément, Eric De la Clergerie, and Lionel Net. 2005. Maf: a morphosyntactic annotation framework. find
  40. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. find
  41. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  42. Alexis CONNEAU and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  43. Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017. Very deep convolutional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1107–1116, Valencia, Spain. Asso… our card
  44. Marta R. Costa-jussà, David Aldón, and José A. R. Fonollosa. 2017. Chinese–spanish neural machine translation enhanced with character and word bitmap fonts. Machine Translation, 31(1):35–47. find
  45. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages… find
  46. Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21–30. find
  47. Mathias Creutz and Krista Lagus. 2005. Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR’05), volume 1, pages 51–59. find
  48. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Trans. Speech Lang. Process., 4(1). find
  49. Falcon Dai and Zheng Cai. 2017. Glyph-aware embedding of Chinese characters. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 64–69, Copenhagen, Denmark. Association for Computational Linguistics. find
  50. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  51. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 204–213, Dublin, Ireland. European Association for Machine Translation. find
  52. M. Domingo, M. Garcıa-Martınez, A. Helle, F. Casacuberta, and M. Herranz. 2018. How Much Does Tokenization Affect Neural Machine Translation? arXiv e-prints. find
  53. Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2018. How much does tokenization affect neural machine translation? arXiv preprint arXiv:1812.08621. — [arXiv:1812.08621](https://arxiv.org/abs/1812.08621) find
  54. Bonaventure F. P. Dossou and Chris C. Emezue. 2021. Crowdsourced phrase-based tokenization for low-resourced neural machine translation: The case of fon language. find
  55. Yerai Doval and Carlos Gómez-Rodríguez. 2019. Comparing neural- and n-gram-based language models for word segmentation. Journal of the Association for Information Science and Technology, 70(2):187–197. find
  56. C. M. Downey, Fei Xia, Gina-Anne Levow, and Shane Steinert-Threlkeld. 2021. A masked segmental language model for unsupervised natural language segmentation. find
  57. Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. find
  58. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics… find
  59. Salah El Hihi and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependencies. In Proceedings of the 8th International Conference on Neural Information Processing Systems, NIPS’95, page 493–499, Cambridge, MA, USA. MIT Press. find
  60. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  61. Micha Elsner, Sharon Goldwater, Naomi Feldman, and Frank Wood. 2013. A joint learning model of word segmentation, lexical acquisition, and phonetic variability. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 42–54, Seattle, Washington, USA. Associati… find
  62. Philip Gage. 1994. A new algorithm for data compression. C Users J., 12(2):23–38. find
  63. Marco Gaido, Beatrice Savoldi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. How to split: the effect of word segmentation on gender bias in speech translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3576–3589, Online. Association for Computati… find
  64. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… find
  65. Yingqiang Gao, Nikola I Nikolov, Yuhuang Hu, and Richard HR Hahnloser. 2020. Character-level translation with self-attention. arXiv preprint arXiv:2004.14788. find
  66. John Goldsmith. 2001. Unsupervised learning of the morphology of a natural language. Computational linguistics, 27(2):153–198. find
  67. S. Goldwater, T. L. Griffiths, and M. Johnson. 2006a. Contextual dependencies in unsupervised word segmentation. In Proc. of COLING-ACL. find
  68. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2009. A Bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112(1):21–54. find
  69. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2011. Producing power-law distributions and damping word frequencies with two-stage language models. Journal of Machine Learning Research, 12(Jul):2335–2382. find
  70. Sharon Goldwater, Mark Johnson, and Thomas Griffiths. 2006b. Interpolating between types and tokens by estimating power-law generators. In Advances in Neural Information Processing Systems, volume 18. MIT Press. find
  71. Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955–3964, Online. Association for Computational Linguistics. our card
  72. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. In International Conference on Learning Representations 2016. find
  73. Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, and Armand Joulin. 2019. Training hybrid language models by marginalizing over segmentations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1477–1482, Florence, Italy. Association for Computati… find
  74. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  75. Stig-Arne Grönroos, Sami Virpioja, and Mikko Kurimo. 2020. Morfessor EM+Prune: Improved subword segmentation with expectation maximization and pruning. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3944–3953, Marseille, France. European Language Resources Association… find
  76. Stig-Arne Grönroos, Sami Virpioja, Peter Smit, and Mikko Kurimo. 2014. Morfessor FlatCat: An HMM-based method for unsupervised and semi-supervised learning of morphology. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1177–1185. find
  77. Rohit Gupta, Laurent Besacier, Marc Dymetman, and Matthias Gallé. 2019. Character-based nmt with transformer. arXiv preprint arXiv:1911.04997. — [arXiv:1911.04997](https://arxiv.org/abs/1911.04997) find
  78. Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozinova, and Tanja Samardzic. 2021. From characters to words: the turning point of BPE merges. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3454–3468, Online. Associ… find
  79. Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020. Dynamic programming encoding for subword segmentation in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042–3051, Online. Association for Computational Linguisti… find
  80. James Henderson. 2020. The unstoppable rise of computational linguistics in deep learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6294–6306, Online. Association for Computational Linguistics. find
  81. Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic tokenization with a language model for neural text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620–1629, Florence, Italy. Association for Computational Lingui… find
  82. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2020. Optimizing word segmentation for downstream task. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1341–1351, Online. Association for Computational Linguistics. find
  83. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2021. Joint optimization of tokenization and downstream model. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 244–255, Online. Association for Computational Linguistics. find
  84. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… our card
  85. Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards neural phrase-based machine translation. In International Conference on Learning Representations. find
  86. Matthias Huck, Simon Riess, and Alexander Fraser. 2017. Target-side word segmentation strategies for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 56–67. find
  87. Gérard Huet. 2003. Lexicon-directed segmentation and tagging of sanskrit. In in « XIIth World Sanskrit Conference, pages 307–325. find
  88. Kyuyeon Hwang and Wonyong Sung. 2017. Character-level language modeling with hierarchical recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 5720– 5724. IEEE. find
  89. Itay Itzhak and Omer Levy. 2021. Models in a spelling bee: Language models implicitly learn the character composition of tokens. find
  90. Miguel Ángel Jiménez-Montaño. 1984. On the syntactic structure of protein sequences and the concept of grammar complexity. Bulletin of Mathematical Biology, 46(4):641–659. find
  91. Mark Johnson and Sharon Goldwater. 2009. Improving nonparameteric Bayesian inference: experiments on unsupervised word segmentation with adaptor grammars. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Ling… find
  92. Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. 2007. Adaptor grammars: A framework for specifying compositional nonparametric Bayesian models. In Advances in Neural Information Processing Systems 19, pages 641– 648. MIT Press. find
  93. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Ass… find
  94. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. find
  95. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. find
  96. Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the hierarchical multiscale lstm. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3215–3227. Association for Computational Linguistics. find
  97. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of the 55th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1492– 1502, Vancouver, Canada. Association for Computational Linguistics. find
  98. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6429–6441, Florence, Italy. Association for Computational Linguisti… find
  99. Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. 2021. How bpe affects memorization in transformers. find
  100. Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. 2016. Character-aware neural language models. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). our card
  101. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  102. Philipp Koehn and Kevin Knight. 2003. Empirical methods for compound splitting. In 10th Conference of the European Chapter of the Association for Computational Linguistics, Budapest, Hungary. Association for Computational Linguistics. our card
  103. Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Segmental recurrent neural networks. In International Conference on Learning Representations. find
  104. Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. 2014. A clockwork rnn. In Proceedings of Machine Learning Research, volume 32, pages 1863–1871, Bejing, China. PMLR. find
  105. Julia Kreutzer and Artem Sokolov. 2018. Learning to Segment Inputs for NMT Favors Character-Level Processing. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT). find
  106. Amrith Krishna, Pavan Kumar Satuluri, and Pawan Goyal. 2017. A dataset for Sanskrit word segmentation. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 105–114, Vancouver, Canada. Association for Computa… find
  107. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… our card
  108. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… our card
  109. Mikko Kurimo, Sami Virpioja, Ville Turunen, and Krista Lagus. 2010. Morpho Challenge 2005-2010: Evaluations and Results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pages 87–95, Uppsala, Sweden. Association for Computational Linguis… find
  110. N Jesper Larsson and Alistair Moffat. 2000. Offline dictionary-based compression. Proceedings of the IEEE, 88(11):1722–1732. find
  111. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5(0):365–378. our card
  112. Jindřich Libovický and Alexander Fraser. 2020. Towards reasonably-sized character-level transformer NMT by finetuning subword systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2572–2579, Online. Association for Computational Linguistic… find
  113. Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2021. Why don’t people use character-level machine translation? find
  114. Constantine Lignos. 2010. Learning from unseen data. In Proceedings of the Morpho Challenge 2010 Workshop, pages 35–38, Helsinki, Finland. Aalto University School of Science and Technology. find
  115. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pro… our card
  116. Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2059–2068, Vancouver, Canada. Association for Computatio… find
  117. Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. Bridging subword gaps in pretrain-finetune paradigm for natural language generation. find
  118. Zhaoxin Luo and Michael Zhu. 2021. Recurrent neural networks with mixed hierarchical structures for natural language processing. CoRR, abs/2106.02562. — [arXiv:2106.02562](https://arxiv.org/abs/2106.02562) find
  119. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany. Association for Computational Linguistics. our card
  120. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Char-BERT: Character-aware Pre-trained Language Model. In COLING. find
  121. Dominik Macháček, Jonáš Vidra, and Ondřej Bojar. 2018. Morphological and language-agnostic word segmentation for nmt. In International Conference on Text, Speech, and Dialogue, pages 277–284. Springer. find
  122. Klaus Macherey, Andrew Dai, David Talbot, Ashok Popat, and Franz Och. 2011. Language-independent compound splitting with morphological operations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1395–1404, Portland, Orego… find
  123. David J. C. MacKay and Linda C. Bauman Peto. 1995. A hierarchical Dirichlet language model. Journal of Natural Language Engineering, 1(3):289–308. find
  124. Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70–74, Online. Association for Computational Lin… find
  125. Carl de Marcken. 1996. Linguistic structure as composition and perturbation. In Proceedings of the 34th Meeting of the Association for Computational Linguistics. find
  126. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. our card
  127. Antonis Maronikolakis, Philipp Dufter, and Hinrich Schütze. 2021. Wine is not v i n. on the compatibility of tokenizations across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2382–2399, Punta Cana, Dominican Republic. Association for Computational Lingui… find
  128. Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using morphological knowledge in open-vocabulary neural language models. In HLT-NAACL. find
  129. Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  130. Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, and Yoshua Bengio. 2017. Multi-scale sequence modeling with a learned dictionary. In MLSLP. find
  131. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic… find
  132. Sabrina J. Mielke and Jason Eisner. 2018. Spell once, summon anywhere: A two-level open-vocabulary language model. CoRR, abs/1804.08205. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) find
  133. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2, pages 3111–3119. our card
  134. Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Čer-nocký. 2012. Subword language modeling with neural networks. preprint (rejected from ICASSP 2012). find
  135. Daichi Mochihashi, Takeshi Yamada, and Naonori Ueda. 2009. Bayesian unsupervised word segmentation with nested pitman-yor language modeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the A… find
  136. Christian Monson, Jaime Carbonell, Alon Lavie, and Lori Levin. 2007. ParaMor: Minimally supervised induction of paradigm structure and morphological analysis. In Proceedings of Ninth Meeting of the ACL Special Interest Group in Computational Morphology and Phonology, pages 117–125, Prague, Czech Rep… find
  137. Christian Monson, Kristy Hollingshead, and Brian Roark. 2009. Probabilistic paramor. In CLEF (Working Notes). Citeseer. find
  138. Amir More, Özlem Çetinoğlu, Çağrı Çöltekin, Nizar Habash, Benoît Sagot, Djamé Seddah, Dima Taji, and Reut Tsarfaty. 2018. CoNLL-UL: Universal morphological lattices for Universal Dependency parsing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC… find
  139. Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2015. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:157–167. find
  140. Vít Novotný, Eniafe Festus Ayetiran, Dávid Lupták, Michal Stefánik, and Petr Sojka. 2021. One size does not fit all: Finding the optimal n-gram sizes for fasttext models across languages. CoRR, abs/2102.02585. — [arXiv:2102.02585](https://arxiv.org/abs/2102.02585) find
  141. Yirong Pan, Xiao Li, Yating Yang, and Rui Dong. 2020. Morphological word segmentation on agglutinative languages for neural machine translation. CoRR, abs/2001.01589. — [arXiv:2001.01589](https://arxiv.org/abs/2001.01589) find
  142. Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks. In AACL/IJCNLP. find
  143. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  144. Yuval Pinter. 2021. Integrating approaches to word representation. find
  145. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102–112, Copenhagen, Denmark. Association for Computational Linguistics. find
  146. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. our card
  147. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin,… our card
  148. M. F. Porter. 1980. An algorithm for suffix stripping. Program, 14(3):130–137. find
  149. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online. Association for Computational Linguistics. our card
  150. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. preprint. find
  151. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  152. Jorma Rissanen. 1989. Stochastic Complexity in Statistical Inquiry Theory. World Scientific Publishing Co., Inc., USA. find
  153. José Carlos Rosales Núñez, Guillaume Wisniewski, and Djamé Seddah. 2021. Noisy UGC translation at the character level: Revisiting open-vocabulary capabilities and robustness of char-based models. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 199–211, Online.… find
  154. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… our card
  155. Benoît Sagot and Pierre Boullier. 2008. SxPipe 2: architecture pour le traitement pré-syntaxique de corpus bruts. Revue TAL, 49(2):155–188. find
  156. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computation… find
  157. Elizabeth Salesky, Andrew Runge, Alex Coda, Jan Niehues, and Graham Neubig. 2018. Optimizing segmentation granularity for neural machine translation. arXiv preprint arXiv:1810.08641. — [arXiv:1810.08641](https://arxiv.org/abs/1810.08641) find
  158. Jonne Saleva and Constantine Lignos. 2021. The effectiveness of morphology-aware segmentation in low-resource neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 164–174, Online.… find
  159. Cicero dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1818–1826. find
  160. Jürgen Schmidhuber. 1991. Neural sequence chunkers. find
  161. Jürgen Schmidhuber. 1992. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234– 242. find
  162. Patrick Schone and Daniel Jurafsky. 2001. Knowledge-free induction of inflectional morphologies. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. our card
  163. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149– 5152. find
  164. Lane Schwartz, Francis Tyers, Lori Levin, Christo Kirov, Patrick Littell, Chi kiu Lo, Emily Prud’hommeaux, Hyunji Hayley Park, Kenneth Steimel, Rebecca Knowles, Jeffrey Micher, Lonny Strunk, Han Liu, Coleman Haley, Katherine J. Zhang, Robbie Jimmerson, Vasilisa Andriyanets, Aldrian Obaja Muis, Naoki… find
  165. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. our card
  166. Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018. Universal word segmentation: Implementation and interpretation. find
  167. Pamela Shapiro and Kevin Duh. 2018. BPE and charcnns for translation of morphology: A cross-lingual comparison and analysis. CoRR, abs/1809.01301. — [arXiv:1809.01301](https://arxiv.org/abs/1809.01301) find
  168. Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2021. ShuoWen-JieZi: Linguistically Informed Tokenizers For Chinese Language Model Pretraining. arXiv, abs/2106.00400. — [arXiv:2106.00400](https://arxiv.org/abs/2106.00400) find
  169. Peter Smit, Sami Virpioja, Stig-Arne Grönroos, and Mikko Kurimo. 2014. Morfessor 2.0: Toolkit for statistical morphological segmentation. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 21–24, Gothenburg, Swe… our card
  170. Matthew Snover and Michael Brent. 2002. A probabilistic model for learning concatenative morphology. Advances in Neural Information Processing Systems, 15:1537–1544. find
  171. Matthew G. Snover and Michael R. Brent. 2001. A Bayesian model for morpheme and paradigm identification. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 490–498, Toulouse, France. Association for Computational Linguistics. find
  172. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast Word-Piece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  173. James A. Storer and Thomas G. Szymanski. 1982. Data compression via textual substitution. J. ACM, 29(4):928–951. find
  174. Baohua Sun, Lin Yang, Catherine Chi, Wenhan Zhang, and Michael Lin. 2019. Squared english word: A method of generating glyph to use super characters for sentiment analysis. In Proceedings of the 2nd Workshop on Affective Content Analysis (AffCon 2019) co-located with the Thirty-Third AAAI Conference… find
  175. Baohua Sun, Lin Yang, Patrick Dong, Wenhan Zhang, Jason Dong, and Charles Young. 2018. Super characters: A conversion from sentiment classification to image classification. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 309–… find
  176. Zhiqing Sun and Zhi-Hong Deng. 2018. Unsupervised neural word segmentation for Chinese via segmental language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4915– 4920, Brussels, Belgium. Association for Computational Linguistics. find
  177. Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML ’11, pages 1017–1024, New York, NY, USA. ACM. find
  178. Dan Tito Svenstrup, Jonas Hansen, and Ole Winther. 2017. Hash embeddings for efficient word representations. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  179. Samson Tan, Shafiq Joty, Lav Varshney, and Min-Yen Kan. 2020. Mind your inflections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5647–5663, Online. Association for Com… find
  180. Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. our card
  181. Yee Whye Teh. 2006. A hierarchical bayesian language model based on Pitman-Yor processes. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 985–992, Sydney, Australia. Association for Computational Linguistics. our card
  182. David Vilar and Marcello Federico. 2021. A statistical extension of byte-pair encoding. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 263–275, Bangkok, Thailand (online). Association for Computational Linguistics. find
  183. Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. 2011. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45–90. find
  184. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2019. Neural machine translation with byte-level subwords. our card
  185. Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. 2017. Sequence modeling via segmentations. In International Conference on Machine Learning, pages 3674–3683. find
  186. Haohan Wang, Peiyan Zhang, and Eric P Xing. 2020a. Word shape matters: Robust machine translation with visual embedding. arXiv preprint arXiv:2010.09997. — [arXiv:2010.09997](https://arxiv.org/abs/2010.09997) find
  187. Lihao Wang, Zongyi Li, and Xiaoqing Zheng. 2021a. Unsupervised word segmentation with bi-directional neural language model. CoRR, abs/2103.01421. — [arXiv:2103.01421](https://arxiv.org/abs/2103.01421) find
  188. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021b. Multi-view subword regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473–482, Online. Association for Computational Linguistic… find
  189. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020b. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. find
  190. Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In COL-ING 1992 Volume 4: The 14th International Conference on Computational Linguistics. our card
  191. J Gerard Wolff. 1975. An algorithm for the segmentation of an artificial language analogue. British Journal of PsychologyJ, 66:79–90. find
  192. F. Wood, J. Gasthaus, C. Archambeau, L. James, and Y.W. Teh. 2011. The sequence memoizer. Communications of the ACM, 54(2):91–98. find
  193. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,… our card
  194. Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021. Vocabulary learning via optimal transport for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… find
  195. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. find
  196. David Yarowsky and Richard Wicentowski. 2000. Minimally supervised morphological analysis by multimodal alignment. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 207–216, Hong Kong. Association for Computational Linguistics. find
  197. Xinsong Zhang, Pengshuai Li, and Hang Li. 2021. AMBERT: A pre-trained language model with multi-grained tokenization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 421–435, Online. Association for Computational Linguistics. find
  198. Giulio Zhou. 2018. Morphological zero-shot neural machine translation. Master’s thesis, University of Edinburgh. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/