Elamur Research⌕
☰
home / papers / 2019 / Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation

Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation

Paper · 2019 · Vladimir Karpukhin, Omer Levy, Jacob Eisenstein et al.

Summary

Translation systems trained on clean text tend to fail on inputs with typos. This holds even for subword (BPE) and character-CNN encoders, because a single misspelled character changes the units these models depend on. Earlier attempts to fix this with synthetic noise only helped on the same artificial noise and hurt clean-text quality. The authors train a Transformer with a CNN character encoder on IWSLT 2016 German, French and Czech→English data. During training, each token is randomly corrupted on the fly by one of four character edits: deletion, insertion, substitution or swap. They then test on real misspellings, drawn from Wikipedia edit histories and corrected Czech essays, which were never used in training. This mixed synthetic noise recovers about 19–54% of the BLEU lost to natural typos and shifts clean-text BLEU by less than one point. Deletions and insertions turn out to matter most. The approach does not help on MTNT Reddit data. This suggests that social-media text differs from standard text mainly in style and vocabulary rather than in spelling errors.

Key points

  • Data and model. About 200K sentence pairs per language pair from IWSLT 2016 (de/fr/cs→en). The model is a base Transformer (6 layers, 512/2048 dimensions, 8 heads) with a Kim et al. (2016) character CNN, 256-dimensional character embeddings and dropout 0.2.
  • Noise schedule. Each training token stays clean with probability 60%. Otherwise it receives one of the four edits, each with probability 10%. Noise is resampled every epoch.
  • Main result. On de-en with maximal natural noise (about 39% of tokens corrupted), BLEU rises from 12.49 to 23.34. Clean-test changes range from −0.67 to +0.33 BLEU.
  • Ablation on de-en.
    • Adding deletion alone gives +4.90 BLEU, and insertion alone gives +2.51.
    • Substitution alone slightly hurts (−0.50).
    • Removing deletion from the full mix costs 8.38 BLEU.
  • MTNT results. Noise training gives no meaningful gain: en-fr goes from 21.1 to 20.6 and fr-en from 23.6 to 24.1. Training on half of the natural typo set also behaved about the same as using no noise.

Interesting ideas

  • I1. Keep training and test noise separate. Train with cheap, generic synthetic noise and evaluate only on real, held-out errors. This gives an honest estimate of robustness without requiring a language-specific error corpus, which is useful for low-resource languages such as Ukrainian.
  • I2. Include edits that change word length. Insertions and deletions shift character n-gram positions and word length, which stresses character encoders more than swaps do. Any noise-augmentation scheme for character or subword models should probably include them.
  • I3. Report a "% recovered" metric. This measures the share of the noise-induced BLEU drop that a method wins back. It makes robustness gains comparable across languages and noise levels better than raw score differences do.
  • I4. Check how well synthetic noise covers real errors. Measure what fraction of real errors a single synthetic edit could produce, then compare that with the gains. This is a quick diagnostic for whether a noise model fits the target error distribution, and it also flagged that random substitutions poorly mimic real ones.
  • I5. Find out what kind of "noise" a domain really contains before augmenting. The MTNT result shows that social-media text mainly differs through slang, code-switching and stylistic shortenings rather than typos. The augmentation should therefore target those phenomena instead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations🏢MetaMenlo Park · United States158 works · 211,459 citations🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations🏢Meta (Israel)Tel Aviv · Israel36 works · 31,506 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Vladimir Karpukhin, Omer Levy, Jacob Eisenstein and Marjan Ghazvininejad. 2019. Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5506

@inproceedings{karpukhin2019training,
  title = {{Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation}},
  author = {Vladimir Karpukhin, Omer Levy, Jacob Eisenstein and Marjan Ghazvininejad},
  booktitle = {Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-5506},
  url = {https://doi.org/10.18653/v1/d19-5506}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223. find
  2. Luz Rello and Ricardo A Baeza-Yates. 2012. Social media is not that bad! the lexical quality of social media. In ICWSM. find
  3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In ICLR. find
  4. Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2017. Robsut wrod reocginiton via semi-character recurrent neural network. In AAAI, pages 3281–3287. find
  5. Chris M Bishop. 1995. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116. find
  6. Karel Šebesta, Zuzanna Bedřichová, Kateřina Šormová, Barbora Štindlová, Milan Hrdlička, Tereza Hrdličková, Jiřı́ Hana, Vladimı́r Petkevič, Tomáš Jelı́nek, Svatava Škodová, Petr Janeš, Kateřina Lundáková, Hana Skoumalová, Šimon Sládek, Piotr Pierscieniak, Dagmar Toufarová, Mi… find
  7. Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2016. find
  8. The IWSLT 2016 evaluation campaign. In International Workshop on Spoken Language Translation. find
  9. Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756– 1766, Melbourne, Australia. Association for Computational… our card
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. our card
  11. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan. find
  12. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, Melbourne, Australia. Association for Comput… our card
  13. Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. arXiv preprint arXiv:1902.09508. find
  14. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 conference of the North American Chapter of the association for computational linguistics: Human language technologies, pages 359–369. our card
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. our card
  16. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. our card
  17. Stefan Wager, Sida Wang, and Percy S Liang. 2013. Dropout training as adaptive regularization. In Advances in neural information processing systems, pages 351–359. find
  18. Georg Heigold, Günter Neumann, and Josef van Genabith. 2017. How robust are character-based word embeddings in tagging and mt against wrod scramlbing or randdm nouse? arXiv preprint arXiv:1704.04441. find
  19. Torsten Zesch. 2012. Measuring contextual fitness using error contexts extracted from the wikipedia revision history. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 529–538. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/karpukhin-2019-training-synthetic-noise-improves-robustness/