Elamur Research⌕
☰
home / papers / 2024 / Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

Paper · 2024 · Lukas Edman, Gabriele Sarti, Antonio Toral et al.

Summary

Byte- and character-level language models can match subword models on many NLP tasks, but their value for neural machine translation under the pretrain-then-finetune approach had barely been tested. This study compares ByT5, a byte-level model, with its subword counterpart mT5. The authors fine-tuned 162 models, varying model size (small, base and large, up to 1.23B parameters), language pair and amount of parallel data. Training data came from NewsCommentary v16 and WMT14 German–English. Evaluation used FLoRes-200, which also allowed zero-shot tests across 200 source languages. ByT5 usually translates better, and its advantage is largest when fine-tuning data is scarce. It also transfers better across languages on average and handles rare and orthographically similar words more accurately. An attribution analysis suggests that ByT5 internally attends to whole words, not only to individual bytes. The cost is speed: ByT5 trains and decodes several times more slowly. The authors therefore recommend byte-level models for low-resource or non-time-critical translation.

Key points

  • Data and settings: Fine-tuning sizes ranged from 0.4k to 250k sentence pairs (NewsCommentary v16), with additional runs on 1.25M and 4.5M pairs (WMT14 German–English). Directions covered {German, Russian}↔English and {Portuguese, English}→Spanish, and FLoRes-200 served for development and testing.
  • Cross-lingual transfer: ByT5 generalizes better to unseen source languages on average. However, its zero-shot quality drops faster than mT5's when it is fine-tuned on many examples.
  • Copying proper nouns: On a control set targeting proper nouns, large models trained on 400 German→English pairs differed sharply. ByT5 copied correctly about 91% of the time, compared with about 25% for mT5. mT5's chrF++ fell by as much as 8 points on the modified test set, while ByT5's chrF++ stayed unchanged.
  • Rare words: With 10k training pairs, ByT5-large had higher word-level accuracy at every frequency level for zero-shot languages. For German, it was ahead at all levels except the most frequent words. The gap narrows as word frequency rises.
  • Efficiency: On a single V100 GPU, ByT5 was roughly 4–5× slower in training and 5–6× slower in inference. It needed fewer epochs to converge, but not enough to offset the slower speed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands70 works · 5,142 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,087 works

Cite

Cite as Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan van Noord and Arianna Bisazza. 2024. Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00651

@article{edman2024are,
  title = {{Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation}},
  author = {Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan {van Noord} and Arianna Bisazza},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00651},
  url = {https://doi.org/10.1162/tacl_a_00651}
}

References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295–4305, Brussels, Belgium. Asso…
  2. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. https://do… find
  3. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693–1703, Berlin, Germany. Associ…
  4. Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. 2019. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  5. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91. https://doi.org/10.1162/tacl a 00448 ↗
  6. Marta R. Costa-jussà, Carlos Escolano, and José A. R. Fonollosa. 2017. Byte-based neural machine translation. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 154–158, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org/10.18653 /… find
  7. Nadir Durrani, Hassan Sajjad, Alexander Fraser, and Helmut Schmid. 2010. Hindi-to-Urdu machine translation through transliteration. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 465–474, Uppsala, Sweden. Association for Computational Linguistics. find
  8. Lukas Edman, Antonio Toral, and Gertjan van Noord. 2022. Subword-delimited downsampling for better character-level translation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 981–992, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https:/… find
  9. Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361, Berlin, Germany. Association for Computational Linguistics. https://d…
  10. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics…
  11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  12. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Confere…
  13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. 2016. Character-aware neural language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. https://doi.org/10.1609/aaai.v30i1.10362 ↗
  14. Antonio Manuel Larriba Flor. 2017. Traducción automática basada en caracteres y redes neuronales. find
  15. Mozhdeh Gheini, Xiang Ren, and Jonathan May. 2021. Cross-attention is all you need: Adapting pretrained Transformers for machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1754–1765, Online and Punta Cana, Dominican Republic. Associa… find
  16. B. Hatim and J. Munday. 2004. Translation: An Advanced Resource Book. Routledge. https://doi.org/10.4324/9780203501887 — [doi:10.4324/9780203501887](https://doi.org/10.4324/9780203501887) ↗
  17. Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019. Towards understanding neural machine translation with word importance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  18. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365–378. https://doi.org/10.1162/tacl a 00067 ↗
  19. Jiahuan Li, Yutong Shen, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2021. When is char better than subword: A systematic study of segmentation algorithms for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio… find
  20. Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2022. Why don’t people use character-level machine translation? In Findings of the Association for Computational Linguistics: ACL 2022, pages 2470–2485, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/10 .18653/v1… find
  21. Digvijay Ingle, Rishabh Tripathi, Ayush Kumar, Kevin Patel, and Jithendra Vepa. 2022. Investigating the characteristics of a transformer in a few-shot setup: Does freezing layers in RoBERTa help? In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,… find
  22. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. https://doi.org/10.1162 /tacl_a_0034… find
  23. Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55(8). https://doi.org/10.1145/3546577 tional Linguistics. https://doi.org/10 .18653/v1/2023.acl-long.689 ↗
  24. Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar van der Wal, Malvina Nissim, and Arianna Bisazza. 2023. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
  25. NLLB Team, Marta Ruiz Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, et al. 2022. No language left behind: Scaling human-centered machine translation. ArXiv, abs/2207.04672. https://doi.org/10.48550/arXiv.2207.04672 ↗
  26. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-4770 ↗
  27. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382, Valencia…
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  30. Ricardo Rei, José G. C. De Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pag…
  31. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR. find
  32. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. https://doi.org /10.18… ↗
  33. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations (ICLR 2014): Workshop Track Proceedings, Banff, AB, Canada. https://doi.org/10 .48550…
  34. Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. 2023. Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  35. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  36. Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. find
  37. Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. Charformer: Fast character transformers via gradient-based subword tokenization. In Proceedings of the Tenth International Conference on Learning Representat…
  38. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185. find
  39. Jörg Tiedemann and Preslav Nakov. 2013. Analyzing the use of character-level translation with sparse and noisy datasets. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 676–684, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA. find
  40. A Hyperparameters We train models using the AdaFactor (Shazeer and Stern, 2018) optimizer with 4000 warmup batches and a final constant learning rate of 1e-4. We batch by \# of tokens (20k for ByT5, 5k for mT5) to provide each model with a roughly equivalent amount of context, as we estimate roughly… find
  41. Elena Voita, Rico Sennrich, and Ivan Titov. 2021b. Language modeling, lexical translation, reordering: The training process of NMT through the lens of classical SMT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8478–8491, Online and Punta Cana, Dom… find
  42. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. https://doi.org/10.1162/tacl a… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/edman-2024-are-character-level-translations-worth/