Elamur Research⌕
☰
home / papers / 2024 / MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization

Paper · 2024 · Orevaoghene Ahia, Sachin Kumar, Hila Gonen et al.

Summary

Byte-level and subword language models usually split non-Latin-script text into many more pieces than English. This raises compute cost and hurts model quality for many languages. Earlier gradient-based tokenizers inside hourglass transformers learn segment boundaries end-to-end. However, they use a single boundary predictor with one target compression rate, so scripts that need more UTF-8 bytes for the same content remain over-fragmented.

The authors propose MAGNET. It keeps the hourglass design (a byte encoder, a pooled middle transformer and an upsampling block) but routes each byte sequence to a boundary predictor dedicated to its script, such as Latin, Cyrillic or Indic. Each predictor is trained with hard Gumbel-sigmoid reparameterization and has its own compression setting.

Experiments on nine typologically diverse languages compare MAGNET with BPE, uncompressed byte-level models and the single-predictor DTP approach. MAGNET produces segments of similar granularity across scripts, speeds up language modelling and keeps downstream performance competitive, with some trade-offs from strong compression of non-Latin languages. For Cyrillic-script work, including Ukrainian, this points toward cheaper and fairer tokenizer-free models.

Key points

  • The motivating disparity is large. An English sentence takes 43 UTF-8 bytes while its Telugu translation takes 148, and non-Latin scripts can need up to about 4× more bytes for the same information.
  • Each boundary predictor is an MLP with a sigmoid, applied over byte hidden states. A Binomial-prior regularizer with hyperparameter β keeps compression near 1/β, and MAGNET assigns a separate predictor and β to each script.
  • The baselines are BPE, byte-level models without compression and DTP (the gradient-based method of Nawrot et al.). Downstream evaluation includes the SIB dataset.
  • Fine-tuning on downstream tasks barely changes the learned boundaries. The few small shifts showed no clear effect on task performance.
  • The paper appeared at NeurIPS 2024. Code and data are released at github.com/orevaahia/magnet-tokenization, and the qualitative examples include Russian, Ukrainian, Hindi and Telugu.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksBias, fairness & safety · 517 worksTokenization & segmentation · 119 works

Cite

Cite as Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov and Noah A. Smith. 2024. MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization. arXiv:2407.08818.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.08818 · arXiv 2407.08818

@misc{ahia2024magnet,
  title = {{MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization}},
  author = {Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov and Noah A. Smith},
  year = {2024},
  doi = {10.48550/arxiv.2407.08818},
  eprint = {2407.08818},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.08818}
}

References

48 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Ács. Exploring bert’s vocabulary. Blog Post, 2019. find
  2. D. Adelani, H. Liu, X. Shen, N. Vassilyev, J. Alabi, Y. Mao, H. Gao, and E.-S. Lee. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Y. Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter of the… find
  3. O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. Mortensen, N. Smith, and Y. Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,…
  4. R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones. Character-level language modeling with deeper self-attention. In AAAI Conference on Artificial Intelligence, 2018. URL https://api.semanticscholar.org/CorpusID:52004855. ↗
  5. C. Arnett, P. D. Riviere, T. A. Chang, and S. Trott. Different tokenization schemes lead to comparable performance in spanish number agreement. ArXiv, abs/2403.13754, 2024. URL https://api.semanticscholar.org/CorpusID:268536952. ↗
  6. M. Artetxe, S. Ruder, and D. Yogatama. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856, 2019.
  7. G. Castillo-lópez, A. Riabi, and D. Seddah. Analyzing zero-shot transfer scenarios across Spanish variants for hate speech detection. In Y. Scherrer, T. Jauhiainen, N. Ljubešić, P. Nakov, J. Tiedemann, and M. Zampieri, editors, Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (Var… find
  8. H. W. Chung, D. Garrette, K. C. Tan, and J. Riesa. Improving multilingual models with language-clustered vocabularies. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546, Online, Nov. 2020… find
  9. J. H. Clark, D. Garrette, I. Turc, and J. Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91, 2022. doi: 10.1162/tacl_a_00448. URL https://aclanthology.org/2022. tacl-1.5. ↗
  10. A. CONNEAU and G. Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_fil… ↗
  11. A. Conneau, R. Rinott, G. Lample, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. find
  12. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Asso… find
  13. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu…
  14. W. Fleshman and B. V. Durme. Toucan: Token-aware character level language modeling. ArXiv, abs/2311.08620, 2023. URL https://api.semanticscholar.org/CorpusID: 265213263. — [arXiv:2311.08620](https://arxiv.org/abs/2311.08620) ↗
  15. N. Godey, R. Castagné, É. de la Clergerie, and B. Sagot. MANTa: Efficient gradient-based tokenization for end-to-end robust language modeling. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2859–2870, Abu Dhabi, United… find
  16. N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538, 2022. doi: 10.1162/tacl_… find
  17. J. Hayase, A. Liu, Y. Choi, S. Oh, and N. A. Smith. Data mixture inference: What do bpe tokenizers reveal about their training data?, 2024. URL https://arxiv.org/abs/2407.16607. — [arXiv:2407.16607](https://arxiv.org/abs/2407.16607) ↗
  18. D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus), 2023. find
  19. J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080, 2020.
  20. E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum? id=rkE3y85ee. ↗
  21. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106. ↗
  22. T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, Jul…
  23. D. Liang, H. Gonen, Y. Mao, R. Hou, N. Goyal, M. Ghazvininejad, L. Zettlemoyer, and M. Khabsa. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language…
  24. T. Limisiewicz, D. Malkin, and G. Stanovsky. You can have your data and balance it too: Towards balanced and efficient multilingual models. In L. Beinborn, K. Goswami, S. Muradoğlu, A. Sorokin, R. Kumar, A. Shcherbakov, E. M. Ponti, R. Cotterell, and E. Vylomova, editors, Proceedings of the 5th Work… find
  25. T. Limisiewicz, T. Blevins, H. Gonen, O. Ahia, and L. Zettlemoyer. Myte: Morphology-driven byte encoding for better and fairer multilingual language modeling. ArXiv, abs/2403.10691, 2024. URL https://api.semanticscholar.org/CorpusID:268512851. ↗
  26. J. Lotz, E. Salesky, P. Rust, and D. Elliott. Text rendering strategies for pixel language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10155–10172, Singapore, Dec. 2023. Association for Computatio… find
  27. C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=S1jE5L5gl. ↗
  28. S. J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W. Y. Lee, B. Sagot, and S. Tan. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. ArXiv, abs/2112.10508, 2021. URL https://api. semanticscholar.org/CorpusID:245335281…
  29. B. Muller, A. Anastasopoulos, B. Sagot, and D. Seddah. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, edito…
  30. P. Nawrot, S. Tworkowski, M. Tyrolski, L. Kaiser, Y. Wu, C. Szegedy, and H. Michalewski. Hierarchical transformers are more efficient language models. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 1559–157… find
  31. P. Nawrot, J. Chorowski, A. Lancucki, and E. M. Ponti. Efficient transformers with dynamic token pooling. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6403–6417, Toronto,… find
  32. P. J. Ortiz Su’arez, B. Sagot, and L. Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim, 2019. Leibniz-Insti… — [doi:10.14618/ids-pub-9021](https://doi.org/10.14618/ids-pub-9021) find
  33. A. Petrov, E. La Malfa, P. H. S. Torr, and A. Bibi. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.15425. ↗
  34. J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder. UNKs everywhere: Adapting multilingual language models to new scripts. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online a…
  35. C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2019. URL https://api.semanticscholar.org/CorpusID:204838007. ↗
  36. P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1…
  37. P. Rust, J. F. Lotz, E. Bugliarello, E. Salesky, M. de Lhoneux, and D. Elliott. Language modelling with pixels. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FkSp8VW8RjH. ↗
  38. E. Salesky, N. Verma, P. Koehn, and M. Post. Multilingual pixel representations for translation and effective cross-lingual transfer. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13845–13861, Singapore, De… find
  39. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In K. Erk and N. A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, Aug. 2016. Associa…
  40. X. Song, A. Salcianu, Y. Song, D. Dopson, and D. Zhou. Fast WordPiece tokenization. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic, Nov.… find
  41. Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JtBRnrlOE… ↗
  42. X. Wang, S. Ruder, and G. Neubig. Multi-view subword regularization. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Comp… find
  43. S. Wu and M. Dredze. Are all languages created equal in multilingual BERT? In S. Gella, J. Welbl, M. Rei, F. Petroni, P. Lewis, E. Strubell, M. Seo, and H. Hajishirzi, editors, Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online, July 2020. Association for Compu…
  44. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, P…
  45. L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022. doi: 10.1162/tacl_a_00461. URL https://aclanthology.org/2… ↗
  46. Y. Yang, Y. Zhang, C. Tar, and J. Baldridge. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proc. of EMNLP, 2019.
  47. L. YU, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum? id=JTmO2V9Xpz. ↗
  48. M. Zampieri, P. Nakov, N. Ljubešić, J. Tiedemann, S. Malmasi, and A. Ali, editors. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), Santa Fe, New Mexico, USA, Aug. 2018. Association for Computational Linguistics. URL https://aclanthology.org/W18-… — [ACL W18-3900](https://aclanthology.org/W18-3900/) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahia-2024-magnet-improving-multilingual-fairness-language/