Elamur Research⌕
☰
home / papers / 2022 / Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

Summary

Almost all modern language models first split text into tokens: by rules, manually, or via BPE, WordPiece or SentencePiece. Such segmentation is poorly suited to languages with rich or nonlinear morphology, languages without whitespace, and texts with typos. In addition, a fixed vocabulary ties the model to a single domain. The authors from Google Research propose CANINE, an encoder that reads text as a sequence of Unicode characters, without a tokenizer and without a vocabulary. Instead of an embedding table, characters are encoded via several hash functions. To keep the longer character sequence from slowing the model down, it is downsampled with strided convolutions before the deep transformer, and for tagging tasks it is upsampled back to full length. Pre-training, as in BERT, uses MLM and NSP. For MLM there are two loss variants: purely character-based, or with prediction of subwords, which serve only as a soft inductive bias during training. On the multilingual benchmark TyDi QA, CANINE outperforms a comparable mBERT while having fewer parameters. For languages where standard tokenization performs poorly, this is a practical step toward tokenizer-free models.

Key points

  • On TyDi QA, CANINE outperforms the comparable mBERT model by 5.7 F1 while having fewer parameters.
  • The architecture consists of three stages: convolutional downsampling of the sequence, a deep transformer, and upsampling for sequence-level tasks. For classification, the zeroth output of the encoder is used. The speed is close to that of a standard BERT.
  • Characters are embedded via K hash functions, each with its own embedding matrix, and the results are concatenated. This covers the entire Unicode space: 1,114,112 code points, of which 143,698 are assigned in Unicode 13.0.
  • Among their motivations, the authors give several examples. These are the nonconcatenative morphology of Arabic (the root k-t-b), languages without whitespace (Thai, Chinese), and punctuation characters functioning as letters (Hawaiian, Twi). Another example: mBERT stripped diacritics, which caused tense information to be lost in Spanish and distinct words to merge in Vietnamese.
  • The work appeared in TACL, vol. 10 (2022), pp. 73–91. The code and checkpoints are published on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Language models & pretraining Tokenization & segmentation

Cite

Cite as Jonathan H. Clark, Dan Garrette, Iulia Turc and John Wieting. 2022. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00448 · arXiv 2103.06874

@misc{clark2022canine,
  title = {{Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation}},
  author = {Jonathan H. Clark, Dan Garrette, Iulia Turc and John Wieting},
  year = {2022},
  doi = {10.1162/tacl\_a\_00448},
  eprint = {2103.06874},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00448}
}

References

70 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Terra Blevins and Luke Zettlemoyer. 2019. Better character language modeling through morphology. In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1156 ↗
  2. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL. https://doi.org/10.1162/tacl a 00051 multiling… ↗ our card
  3. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP. https://doi.org/10.18653/v1/2020.findings-emnlp.414 ↗ our card
  4. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In Proceedings of ICLR. find
  5. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. TACL. our card
  6. Jan A. Botha and Phil Blunsom. 2013. Adaptor Grammars for learning non-concatenative morphology. In Proceedings of EMNLP. find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. https://doi.org/10.18653/v1 /2020.acl-m… find
  8. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Junichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of COLING. https://doi.org/10.18653/v1/2020.coling-main.609 ↗
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… our card
  10. Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. 2020. Funnel-Transformer: Filtering out sequential redundancy for efficient language processing. In Proceedings of NeurIPS. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. our card
  12. Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of NAACL. find
  13. Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. TACL. https://doi.org/10.1162/tacl_a_00032 ↗ our card
  14. Jacob Buckman and Graham Neubig. 2018. Neural lattice language models. TACL. https://doi.org/10.1162/tacl a 00036 ↗
  15. Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. 2019. Bridging the gap for tokenizer-free language models. arXiv preprint arXiv:1908.10322. — [arXiv:1908.10322](https://arxiv.org/abs/1908.10322) find
  16. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In Proceedings of NAACL. https://doi.org/10.18653/v1/N16-1155 ↗
  17. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In Proceedings of ICLR. find
  18. Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, and Armand Joulin. 2019. Training hybrid language models by marginalizing over segmentations. In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1143 ↗
  19. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. find
  20. Improving Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  21. System Demonstrations. https://doi.org/10.18653/v1/D18-2012 ↗ our card
  22. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In Proceedings of NeurIPS. find
  23. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2018. Byte-level machine reading across morphologically varied languages. In Proceedings of AAAI. find
  24. Jason Lee, Eth Zürich, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. TACL. https://doi.org/10 .1162/tacl_a_00067 our card
  25. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM). https://doi.org /10.1145/2505515.250… — [doi:10.1145/2505515.2505665](https://doi.org/10.1145/2505515.2505665) ↗
  26. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of ACL. https://doi.org/10.18653/v1/P16-1100 ↗ our card
  27. Austin Matthews, Graham Neubig, and Chris Dyer. 2019. Using morphological knowledge in open-vocabulary neural language models. In Proceedings of NAACL. https://doi.org/10.18653/v1/N18-1130 ↗
  28. Kyuyeon Hwang and Wonyong Sung. 2017. Character-level language modeling with hierarchical recurrent neural networks. In Proceedings of ICASSP. https://doi.org/10 .1109/ICASSP.2017.7953252 find
  29. Sebastian J. Mielke and Jason Eisner. 2019. Spell once, summon anywhere: A two-level open-vocabulary language model. In Proceedings of AAAI. https://doi.org/10 .1609/aaai.v33i01.33016843 find
  30. Prabhu Kaliamoorthi, Sujith Ravi, and Zornitsa Kozareva. 2019. PRADO: Projection attention networks for document classification on-device. In Proceedings of EMNLP. https://doi.org/10.18653/v1/D19-1506 ↗
  31. Rik van Noord, Antonio Toral, and Johan Bos. 2020. Character-level representations improve drs-based semantic parsing even in the age of BERT. In Proceedings of EMNLP. https://doi.org/10.18653/v1/2020.emnlp-main.371 ↗
  32. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of ACL. https://doi.org/10.18653/v1/P17-1137 ↗
  33. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Learning to discover, ground and use words with segmental neural language models. In Proceedings of ACL. find
  34. Brendan O’Connor, Michel Krieger, and David Ahn. 2010. TweetMotif: Exploratory search and topic summarization for twitter introduction and description. In Proceedings of the International AAAI Conference on Web and Social Media. find
  35. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of AAAI. our card
  36. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2021. Random feature attention. In Proceedings of ICLR. find
  37. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of ACL. our card
  38. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL. https://doi.org/10.18653/v1/N18-1202 ↗ our card
  39. Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of EMNLP: Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel R Bowman. 20… find
  40. Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT. arXiv preprint arXiv:2003.04985. — [arXiv:2003.04985](https://arxiv.org/abs/2003.04985) find
  41. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of EMNLP. find
  42. Ilya Sutskever, James Martens, and Geoffrey E. Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of ICML. find
  43. Yuval Pinter, Marc Marone, and Jacob Eisenstein. 2019. Character eyes: Seeing language through character-level taggers. In Proceedings of BlackboxNLP. https://doi.org/10.18653/v1/W19-4811 ↗
  44. Dan Svenstrup, Jonas Meinertz Hansen, and Ole Winther. 2017. Hash embeddings for efficient word representations. In Proceedings of NeurIPS. find
  45. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is Multilingual BERT? In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1493 ↗ our card
  46. David Talbot and John Talbot. 2008. Bloom maps. In Proceedings of the Workshop on Analytic Algorithmics and Combinatorics (ANALCO). https://doi.org/10.1137/1.9781611972986.4 ↗
  47. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-Dropout: Simple and effective subword regularization. In Proceedings of ACL. https://doi.org/10.18653/v1/2020.acl-main.170 ↗ our card
  48. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1452 ↗ our card
  49. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1561 ↗ our card
  50. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL. https://doi.org/10.3115/1118853 .1118877 ↗ our card
  51. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. our card
  52. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of NAACL. https://doi.org/10.3115/1119176 .1119195 ↗ our card
  53. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. https://www.semanticscholar.org/paper/Language -Models-are-Unsupervised-Multitask -Learners-Radford-Wu/9405cc0d61699 88371b2755e573cc… ↗
  54. Clara Vania, Andreas Grivas, and Adam Lopez. 2018. What do character-level models learn about morphology? The case of dependency parsing. In Proceedings of EMNLP. find
  55. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. our card
  56. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of ACL. https://doi.org/10.18653/v1/P17-1184 ↗
  57. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of ACL. https://doi.org/10.18653/v1/P16-1162 ↗ our card
  58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS. our card
  59. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL. our card
  60. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Proceedings of NAACL. https://doi.org/10.18653/v1/2021.naacl-main.40 ↗
  61. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In Proceedings of ICLR. find
  62. Zihan Wang, Karthikeyan K. Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of EMNLP. https://doi.org/10.18653/v1/2020.findings-emnlp.240 ↗
  63. Xiaodong Yu, Stephen Mayhew, Mark Sammons, and Dan Roth. 2018. On the strength of character language models for multilingual named entity recognition. In Proceedings of EMNLP. find
  64. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. In Proceedings of EMNLP. https://doi.org/10.18653/v1/D16-1157 ↗
  65. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for longer sequences. In Proceedings of NeurIPS. our card
  66. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  67. Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in Chinese, English, Japanese and Korean? arXiv preprint arXiv:1708.02657v2. — [arXiv:1708.02657](https://arxiv.org/abs/1708.02657) find
  68. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NeurIPS. our card
  69. Xinsong Zhang and Hang Li. 2021. AMBERT: A pre-trained language model with multi-grained tokenization. In Findings of ACL. https://doi.org/10.18653/v1/2021 .findings-acl.37 ↗
  70. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of EMNLP. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/