Elamur Research⌕
☰
home / papers / 2021 / Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Summary

Most NLP models rely on a subword tokenizer that is fixed before training. This makes them fragile to rare words, typos and noisy text. It also breaks low-resource languages into many fragments and makes adaptation to new domains harder. Feeding raw characters to a Transformer avoids these issues, but the long sequences are costly. The authors propose Gradient-Based Subword Tokenization (GBST), a module that learns segmentation jointly with the rest of the network. It works on byte embeddings in four steps:

  • it applies a 1D convolution;
  • it builds candidate blocks of several lengths by mean pooling;
  • it scores each candidate with a linear layer and mixes them per position using softmax weights;
  • it mean-pools the result to shorten the sequence.

Charformer is a byte-level encoder-decoder Transformer that places GBST in front. A re-scaled variant moves more capacity into the encoder. The authors test it on English GLUE, long-document classification, social-media toxicity detection and multilingual benchmarks such as XNLI and PAWS-X. Charformer beats byte-level baselines and roughly matches subword models, outperforming them on noisy text. It is also notably faster. This suggests that tokenizer-free, fully end-to-end models are becoming practical.

Key points

  • GBST considers blocks of up to M = 4 bytes over a 256-value byte vocabulary. An optional calibration step lets block scores at different positions attend to each other. A Tensorflow reference implementation of the module is provided.
  • Against a ByT5-like byte-level baseline, Charformer uses about half the memory and runs 10–93% faster. It trains 28% faster than mT5 with roughly 3× fewer parameters, at comparable quality.
  • On PAWS-X translate-train-all, Charformer SBase (134M parameters) averages 88.2, or 88.6 with longer pre-training. mT5 Base (582M) scores 89.3 and Byte-level T5 Base scores 87.0.
  • On PAWS-X cross-lingual zero-shot, Charformer SBase reaches 85.2, compared with 81.9 for mBERT Base and 86.4 for mT5 Base.
  • The learned latent subwords are partly interpretable. However, the method assumes subwords are contiguous, so non-concatenative morphology may be hard for it to capture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics Tokenization & segmentation

Cite

Cite as Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu and Donald Metzler. 2021. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. arXiv:2106.12672.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.12672 · arXiv 2106.12672

@misc{tay2021charformer,
  title = {{Charformer: Fast Character Transformers via Gradient-based Subword Tokenization}},
  author = {Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu and Donald Metzler},
  year = {2021},
  doi = {10.48550/arxiv.2106.12672},
  eprint = {2106.12672},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.12672}
}

References

68 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020, 2020. URL http://arxiv.org/abs/1910.11856. https://github.com/tensorflow/mesh https://github.com/google-research/text-to-text-transfer-transformer 8 htt… ↗ our card
  2. Yonatan Belinkov and Yonatan Bisk. Synthetic and Natural Noise Both Break Neural Machine Translation. In Proceedings of ICLR 2018, 2018. URL http://arxiv.org/abs/1711.02173. ↗
  3. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, abs/1903.04561, 2019. URL http://arxiv.org/abs/1903.04561. — [arXiv:1903.04561](https://arxiv.org/abs/1903.04561) ↗
  4. Kaj Bostrom and Greg Durrett. Byte Pair Encoding is Suboptimal for Language Model Pretraining. In Findings of EMNLP 2020, pages 4617–4624, 2020. doi: 10.18653/v1/2020.findings-emnlp.414. our card
  5. Manuel Carreiras, Margaret Gillon-Dowens, Marta Vergara, and Manuel Perea. Are vowels and consonants processed differently? event-related potential evidence with a delayed letter paradigm. Journal of Cognitive Neuroscience, 21(2):275–288, 2008. find
  6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. our card
  7. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020. find
  8. Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking Embedding Coupling in Pre-trained Language Models. In Proceedings of ICLR 2021, 2021. our card
  9. Jon Clark, Tom Kwiatkowski, Jennimaria Palomaki, Michael Collins, and Dan Garrette. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. In Transactions of the ACL, 2020. our card
  10. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation. arXiv preprint arXiv:2103.06874, 2021. our card
  11. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating Cross-lingual Sentence Representations. In Proceedings of EMNLP 2018, 2018. URL http://arxiv.org/abs/1809.05053. ↗
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Comp… find
  13. Jacob Devlin. Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the cpu. arXiv preprint arXiv:1705.01991, 2017. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL 2019, 2019. URL http://arxiv.org/abs/1810.04805. ↗ our card
  15. William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. our card
  16. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics, page… find
  17. Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. FRAGE: Frequency-Agnostic Word Representation. In Proceedings of NIPS 2018, 2018. find
  18. Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  19. Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. Dynamic programming encoding for subword segmentation in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042–3051, Online, July 2020. Association for Computational Ling… find
  20. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. Optimizing word segmentation for downstream task. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1341–1351, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v… find
  21. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. Joint Optimization of Tokenization and Downstream Model. In Findings of ACL-IJCNLP 2021, 2021. URL http://arxiv.org/abs/2105.12410. ↗
  22. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018, 2018. URL http://arxiv.org/abs/1801.06146. ↗
  23. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. In Proceedings of ICML 2020, 2020. our card
  24. Kyuyeon Hwang and Wonyong Sung. Character-level language modeling with hierarchical recurrent neural networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5720–5724. find
  25. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  26. Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation. In Proceedings of ICLR 2021, 2021. ISBN 0080437516. our card
  27. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6429–6441, Florence, Italy, July 2019. Association for Computational Ling… find
  28. Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. Character-aware neural language models. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016. our card
  29. Julia Kreutzer and Artem Sokolov. Learning to segment inputs for nmt favors character-level processing, 2018. find
  30. Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, July 2018. Association for Computati… our card
  31. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, November 201… our card
  32. Guillaume Lample and Alexis Conneau. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS 2019, 2019. URL https://github.com/google-research/bert. ↗
  33. Hye-Won Lee, Keith Rayner, and Alexander Pollatsek. The relative contribution of consonants and vowels to word identification during reading. Journal of Memory and Language, 44(2):189–205, 2001. find
  34. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365–378, 2017. doi: 10.1162/tacl_a_00067. URL https://aclanthology.org/Q17-1026. ↗ our card
  35. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating Cross-lingual Extractive Question Answering. In Proceedings of ACL 2020, 2020. URL http://arxiv.org/abs/1910.07475. ↗ our card
  36. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. CharBERT: Character-aware pre-trained language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 39–50, Barcelona, Spain (Online), December 2020. International Committee on Computa… find
  37. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150, 2011. our card
  38. Christopher Manning and Hinrich Schütze. Foundations of statistical natural language processing. MIT press, 1999. find
  39. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, 2013. our card
  40. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055–4064. PMLR, 2018. find
  41. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT 2018, 2018. our card
  42. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/… our card
  43. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582–5591, Florence, Italy, July 2019. Association for Computational Linguistics. d… our card
  44. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21, 2020. URL http://arxiv.org/abs/1910.10683. ↗ our card
  45. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November 2016. Association for Computational Ling… our card
  46. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. Xtreme-r: Towards more challenging and nuanced multilingual evaluation. arXiv preprint arXiv:2104.07412, 2021. our card
  47. Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
  48. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computatio… our card
  49. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. find
  50. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. arXiv preprint arXiv:1811.02084, 2018. our card
  51. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642,… find
  52. Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.04985, 2020. — [arXiv:2003.04985](https://arxiv.org/abs/2003.04985) find
  53. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020a. our card
  54. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732, 2020b. our card
  55. Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint arXiv:2105.03322, 2021. — [arXiv:2105.03322](https://arxiv.org/abs/2105.03322) find
  56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S… our card
  57. Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. Sequence modeling via segmentations. In International Conference on Machine Learning, pages 3674–3683. PMLR, 2017. find
  58. Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. find
  59. Xinyi Wang, Sebastian Ruder, and Graham Neubig. Multi-view Subword Regularization. In Proceedings of NAACL 2021, 2021. URL http://arxiv.org/abs/2103.08490. ↗
  60. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… find
  61. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Ni… find
  62. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, pages 1391– 1399, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 978… — [doi:10.1145/3038912.3052591](https://doi.org/10.1145/3038912.3052591) find
  63. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer, 2020. our card
  64. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2105.13626, 2021. URL http://arxiv.org/abs/2105.13626. ↗
  65. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proceedings of EMNLP 2019, 2019. URL http://arxiv.org/abs/1908.11828. ↗ our card
  66. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020. our card
  67. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems, pages 649–657, 2015. URL http://arxiv.org/abs/1509.01626\#. ↗ our card
  68. A PPENDIX 7.1 H YPERPARAMETERS This section describes the hyperparameters that we use in our experiments. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/