Elamur Research⌕
☰
home / papers / 2019 / Multilingual Neural Machine Translation with Language Clustering

Multilingual Neural Machine Translation with Language Clustering

Paper · 2019 · Xu Tan, Jiale Chen, Di He et al.

Summary

Supporting thousands of languages is hard with either extreme: one translation model per language pair is too costly, and a single shared model struggles when its languages are very dissimilar. The authors ask how to decide which languages should share a multilingual NMT model when the number of models is limited. They study English-centric translation in both directions, from 23 languages into English and from English into the same 23, using IWSLT data. They compare two ways of grouping languages, with one model (following Johnson et al., 2017) trained per group:

  • Language family: groups taken from the Ethnologue taxonomy.
  • Learned language embeddings: a single universal model is trained over all languages, and the vectors it learns for its language tags are grouped with hierarchical clustering.

The embedding-based groups mirror finer linguistic structure, such as branches within Indo-European. They also reflect regional and cultural closeness. In most cases they give higher BLEU than family-based grouping, and the authors report gains over the baseline setups for nearly every language. This offers a data-driven way to trade translation quality against compute when deploying multilingual systems.

Key points

  • The data covers 23 languages paired with English, collected from IWSLT campaigns between 2011 and 2018. Training sets range from about 14K sentence pairs (Sl) to 180K. Bg, El, Hu and Ja had no official dev/test sets, so 1K pairs were randomly split off for each.
  • Ethnologue lists 7,472 languages in 152 families, and the 23 languages studied fall into 8 of them. The authors note drawbacks of family-based grouping: language isolates are not covered, and the number of families is high.
  • Language embeddings stabilise early in training, before the universal model reaches its best accuracy. This makes them cheap to obtain.
  • In the English→X direction, the number of clusters was chosen with the elbow method, yielding 5 groups:
    • Romance: Ro, Fr, It, Es, Pt
    • Germanic: De, Nl
    • Mostly Slavic: Sl, Bg, El, Ru, Sk, Cs, Pl
    • Tr, Hu, Zh, Ja
    • Vi, Th, He, Ar, Fa
  • Separate per-language models still perform best on roughly 10 languages in the English→X direction. Clustering helps most for languages with little training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓Peking UniversityBeijing · China100 works · 35,618 citations🎓University of Science and Technology of ChinaHefei · China26 works · 111,012 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 958 worksMultilingual & cross-lingual NLP · 467 works

Cite

Cite as Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin and Tie-Yan Liu. 2019. Multilingual Neural Machine Translation with Language Clustering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1089

@inproceedings{tan2019multilingual,
  title = {{Multilingual Neural Machine Translation with Language Clustering}},
  author = {Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin and Tie-Yan Liu},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1089},
  url = {https://doi.org/10.18653/v1/d19-1089}
}

References

3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate.
  2. Language Name and Code Dik Bakker, André Müller, Viveka Velupillai, Søren Wichmann, Cecil H Brown, Pamela Brown, Dmitry Egorov, Robert Mailhammer, Anthony Grant, and Eric W Holman. 2009. Adding typology to lexicostatistics: A combined approach to language classification. Linguistic Typology, 13(1)… find
  3. The language names and their corresponding language codes according to ISO 639-1 standard are listed in Table 5. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2019-multilingual-neural-machine-translation-language/