Elamur Research⌕
☰
home / papers / 2024 / Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese

Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese

Summary

The paper asks how much tokenisation choices affect English–Maltese machine translation. Maltese is a morphologically rich language with Semitic roots, and generic English-oriented tools often handle its hyphenated articles badly. The authors found that the Maltese side of OPUS-100 contains irregular, non-systematic spacing errors. Models trained on it learn to reproduce these errors, and the extra tokens can push BLEU upward misleadingly. They re-tokenised the Maltese data with the regex-based MLRS tokeniser and released the result as OPUS-100-Fix. Training on the corrected data improved BLEU on the fixed test set by 2.49 points.

They then compared subword tokenisers (BPE, SentencePiece Unigram and SentencePiece BPE), alone and combined with word-level tokenisers (Moses, OpenNMT, MLRS), across several vocabulary sizes. Plain BPE with 8,000 merges worked best. Finally, they compared Transformer base and large models trained from scratch, a fine-tuned mBART-50, and NLLB both out of the box and fine-tuned. Fine-tuned NLLB gave the strongest results. The work shows that preprocessing decisions can noticeably change measured MT quality for low-resource languages.

Key points

  • OPUS-100-Fix: The corrected English–Maltese portion of OPUS-100 is released on Hugging Face (MLRS/OPUS-MT-EN-Fixed). The Maltese side was cleaned with the MLRS tokeniser, and training on it gave +2.49 BLEU over training on the original data.
  • Hyphenated articles: Generic detokenisers split Maltese article hyphens, for example writing "il - kelb" instead of keeping the article attached as in "il-kelb".
  • Vocabulary size (Transformer large, ENG→MLT):
    • BPE: 51.29 BLEU at 8k, 50.00 at 16k, 41.67 at 32k.
    • SentencePiece Unigram collapsed to 2.73 BLEU at 16k.
    • SentencePiece BPE collapsed to 1.39 BLEU at 32k.
  • Combining tokenisers: Adding a word-level tokeniser in front of a subword tokeniser gave no significant benefit.
  • Best model (ENG→MLT): NLLB-FT reached 52.25 BLEU and 76.14 chrF2, and data filtering raised it to 52.65. Out-of-the-box NLLB scored only 39.69 BLEU. Filtering lowered scores for the other models, and noisy-channel reranking gave mixed results.

Interesting ideas

  • I1. Audit the corpus for spacing errors: Before training, check the corpus for irregular spacing and repair it with a language-aware rule tokeniser, because models faithfully copy such noise into their output. — For Mova (tokens): Scan our English training text and treebanks for inconsistent splitting of contractions, hyphenated compounds and possessive 's, and normalise them with the en tokenizer rules before training the tagger.
  • I2. Detokenisation is language-specific: Joining tokens back together needs its own rules (for Maltese, keeping article hyphens attached), not a generic English default. — For Mova (back): Give the graph→text generator explicit, tested rules for reattaching clitics, hyphens and punctuation, so that compressor round-trips do not introduce spacing artefacts.
  • I3. Spurious tokens can inflate n-gram scores: If the reference and the system output are segmented differently, extra tokens can make an n-gram metric look better than the translation really is. — For Mova (learn): Add a negative control to evaluation that re-scores outputs after normalising tokenisation, so gains caused only by segmentation differences are caught.
  • I4. Sweep vocabulary size in low-data settings: Small subword vocabularies won here, and larger ones sometimes caused a sudden collapse rather than a gradual decline. — For Mova (tree): When choosing the subword vocabulary for the gparse transformer parser trained on modest treebanks, run a small sweep from small sizes upward and watch for sharp drops.
  • I5. Filtering can hurt small datasets: Removing data helped only the pretrained model and hurt the models trained from scratch, which the authors link to data scarcity. — For Mova (learn): When filtering silver data by witness-parser agreement, compare scores with and without the filter on small domains, since aggressive pruning may cost more than the noise it removes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation · Semitic languages · Tokenization & segmentation

Institutions

🎓University of MaltaMsida · Malta12 works · 573 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.loresmt-1.11

@inproceedings{abela2024tokenisation,
  title = {{Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese}},
  author = {Abela, Kurt and Micallef, Kurt and Tanti, Marc and Borg, Claudia},
  booktitle = {Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.loresmt-1.11},
  url = {https://doi.org/10.18653/v1/2024.loresmt-1.11}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zaid Alyafeai, Maged S Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2023. Evaluating various tokenizers for arabic text classification. Neural Processing Letters, 55(3):2911–2933. find our card
  2. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  3. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. arXiv preprint arXiv:2107.10821. ↗
  4. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177–180. Association for Computational Linguistics. find
  5. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. Preprint, arXiv:1804.10959. ↗ our card
  6. Marta R. Costa-jussà, James Cross NLLB Team, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. find
  7. Jan Christian Blaise Cruz. 2023. Samsung r&d institute philippines at wmt 2023. arXiv preprint arXiv:2310.16322. ↗
  8. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. arXiv preprint arXiv:1706.09733. ↗ our card
  9. Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2019. How much does tokenization affect neural machine translation? In International Conference on Computational Linguistics and Intelligent Text Processing, pages 545–554. Springer. find
  10. Alexander Erdmann. 2020. Practical Morphological Modeling: Insights from Dialectal Arabic. The Ohio State University. find
  11. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. ↗ our card
  12. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗ our card
  13. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Preprint, arXiv:2001.08210. ↗
  14. Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, et al. 2020. Xlm-t: Scaling up multilingual machine translation with pretrained cross-lingual transformer encoders. arXiv preprint arXiv:2012.15547. ↗
  15. Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics. find
  16. Makoto Morishita, Keito Kudo, Yui Oka, Katsuki Chousa, Shun Kiyono, Sho Takase, and Jun Suzuki. 2022. Nt5 at wmt 2022 general translation task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 318–325. find
  17. Albert Gatt and Slavomír Čéplö. 2013. Digital corpora and other electronic resources for maltese. In Corpus linguistics, pages 96–97. UCREL Lancaster. find
  18. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. ↗ our card
  19. Csaba Oravecz, Katina Bontcheva, David Kolovratník, Bogomil Kovachev, and Christopher Scott. 2022. etranslation’s submissions to the wmt22 general machine translation task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 346–351. find
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  21. Claudia Borg. 2023. Um-dfki maltese speech translation. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 433–441. find our card
  22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  23. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025. ↗ our card
  24. Jitao Xu, Sadaf Abdul Rauf, Minh Quang Pham, and François Yvon. 2021. Lisn@ wmt 2021. In 6th Conference on Statistical Machine Translation. find
  25. Matı̄ss Rikters and Makoto Miwa. 2023. Aist airc submissions to the wmt23 shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 155–161. find
  26. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2019. Masked language model scoring. arXiv preprint arXiv:1910.14659. ↗ our card
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. find our card
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. Preprint, arXiv:1508.07909. ↗ our card
  29. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. ↗
  30. Maali Tars, Taido Purason, and Andre Tättar. 2022. Teaching unseen low-resource languages to large translation models. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 375–380. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find our card
  32. Daimeng Wei, Zongyao Li, Zhanglin Wu, Zhengzhe Yu, Xiaoyu Chen, Hengchao Shang, Jiaxin Guo, Minghan Wang, Lizhi Lei, Min Zhang, et al. 2021. Hw-tsc’s participation in the wmt 2021 news translation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 225–231. find
  33. Aiden Williams, Kurt Abela, Rishu Kumar, Martin Bär, Hannah Billinghurst, Kurt Micallef, Ahnaf Mozib Samin, Andrea DeMarco, Lonneke van der Plas, and Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  34. Jian Yang, Yuwei Yin, Shuming Ma, Dongdong Zhang, Zhoujun Li, and Furu Wei. 2022. Hlt-mt: High-resource language-specific training for multilingual neural machine translation. arXiv preprint arXiv:2207.04906. ↗
  35. Kyra Yee, Yann Dauphin, and Michael Auli. 2019. Simple and effective noisy channel modeling for neural machine translation. In Conference on Empirical Methods in Natural Language Processing. find
  36. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020a. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867. ↗ our card
  37. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020b. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867. ↗ our card
  38. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020c. Improving massively multilingual neural machine translation and zero-shot translation. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abela-2024-tokenisation-machine-translation-does-matter/