Elamur Research⌕
☰
home / papers / 2018 / MTNT: A Testbed for Machine Translation of Noisy Text

MTNT: A Testbed for Machine Translation of Noisy Text

Summary

Modern MT systems, including neural ones, often break down on the informal, error-filled language of social media. Researchers had no public parallel corpus of naturally noisy text, so earlier robustness work relied on synthetic perturbations. Michel and Neubig fill this gap with MTNT, a benchmark built from Reddit comments in English, French and Japanese, which professional translators rendered to and from English.

Noisy comments were chosen automatically. The authors first discarded URLs, bot posts and text in other languages. They then kept comments containing words absent from clean contrast corpora (for English and French) and comments whose lines scored very low under a subword language model.

The resource covers four directions (en-fr, fr-en, en-ja, ja-en) and adds monolingual data for semi-supervised adaptation and noisy language modeling. NMT baselines trained on standard corpora perform poorly on it. Brief fine-tuning on the in-domain training split raises BLEU noticeably, but translations of heavily noisy sentences remain inadequate. MTNT therefore offers a realistic, open testbed for noise-robust translation research.

Key points

  • Dataset size: each language pair has about 1,000 test sentences and roughly 6k–36k training sentences.
  • Contrast corpora:
    • English: the WMT 2017 de-en English side (≈5.85M sentences).
    • French: the WMT 2015 en-fr French side (≈40.86M sentences).
    • Japanese: KFTT, JESC and TED combined (≈4.19M sentences).
  • Noise detection:
    • Text was segmented with sentencepiece BPE, using vocabularies of 1,000 for English and French and 4,000 for Japanese.
    • A 5-gram Kneser-Ney model built with kenLM scored each line.
    • A comment was flagged as noisy if any line fell below the 1st percentile of reference-corpus scores.
  • Fine-tuning: one epoch of SGD on the MTNT training data improved BLEU by 3.17–7.96 points, yet very noisy inputs still failed.
  • Language-modeling baselines:
    • Character-level baselines were a 6-gram KenLM model and AWD-LSTM.
    • AWD-LSTM was better, with test bpc of 1.810 (English), 1.705 (French) and 5.225 (Japanese).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1050

@inproceedings{michel2018mtnt,
  title = {{MTNT: A Testbed for Machine Translation of Noisy Text}},
  author = {Michel, Paul and Neubig, Graham},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1050},
  url = {https://doi.org/10.18653/v1/d18-1050}
}

References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  3. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. How noisy social media text, how diffrnt social media sources? In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 356–364. find our card
  4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. International Conference on Learning Representations. find
  5. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2017. A dataset and classifier for recognizing social Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74… find our card
  6. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), pages 261–268. find
  7. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedin… find
  9. Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. find
  10. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1965–1974. find
  11. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A. Smith. A dependency parser for tweets. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1001–1012. find our card
  12. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–391. Mu Li, Yinggong Zhao, Dongdong Zhang,… find
  13. David Crystal. 2001. Language and the Internet. Cambridge University Press. find
  14. Brenda Danet and Susan Herring. 2007. The Multilingual Internet: Language, Culture, and Communication Online. Oxford University Press., New York. find
  15. Marco Lui and Timothy Baldwin. langid.py: An off-the-shelf language identification tool. In Proceedings of the Association for Computational Linguistics 2012 System Demonstrations, pages 25–30. find our card
  16. Jacob Eisenstein. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369. find our card
  17. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation. find
  18. Lluís Formiga and José A. R. Fonollosa. Dealing with input noise in statistical machine translation. In Proceedings of the Conference on Computational Linguistics 2012: Posters, pages 319–328. find
  19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing lstm language models. International Conference on Learning Representations. find our card
  20. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, an… find
  21. Susan Herring, editor. 2003. Media and Language Change. Special issue of Journal of Historical Pragmatics 4:1. find
  22. Pierre Isabelle, Colin Cherry, and George Foster. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496. find our card
  23. Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. find
  24. Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the Association for Computational Linguistics 2010 Conference Short Papers, pages 220–224. find our card
  25. Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709. find our card
  26. Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt. ↗
  27. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613–624. International World Wide Web Confere… find
  28. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980. find our card
  29. Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable japanese morphological analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT), pages 529–533. find our card
  30. Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1483–1489. find
  31. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. Improved part-of-speech tagging for online conversational text with word clusters. Association for Computational Linguistics. find our card
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  33. Kay Peterson. 2011. Openmt12 evaluation. find
  34. Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. find our card
  35. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. find our card
  36. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. R. Pryzant, Y. Chung, D. Jurafsky, and D. Britz. Jesc: Japanese-engl… find our card
  37. Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2017. Robsut wrod reocginiton via semi-character recurrent neural network. In AAAI, pages 3281–3287. find
  38. Rico Sennrich. How grammatical is character-level neural machine translation? assessing mt quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382. Matthias Sperb… find our card
  39. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/michel-2018-mtnt-testbed-machine-translation-noisy/