Elamur Research⌕
☰
home / papers / 2020 / SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Paper · 2020 · Nathan Ng, Kyunghyun Cho and Marzyeh Ghassemi

Summary

Classifiers and sequence models trained on one domain often perform poorly on out-of-domain (OOD) inputs. For images, simple transformations like rotation keep new examples realistic, but no comparable meaning-preserving operations exist for text. The authors propose SSMBA, a task-agnostic augmentation scheme based on denoising autoencoders. It works in two steps:

  1. It corrupts a training input so that it leaves the data manifold, using the MLM masking procedure.
  2. It restores the input with a pre-trained masked language model, which pulls it back onto the manifold.

The new examples get their labels in one of two ways: they inherit the label of the source example, or a teacher model fit to the unaugmented training set assigns them. The method needs no task knowledge and no dataset-specific fine-tuning. It was tested on sentiment analysis, NLI and machine translation, covering 9 datasets and 4 model types. SSMBA beat both unaugmented baselines and competing augmentation methods, in-domain and OOD. This makes it a cheap, general tool for improving robustness, including in low-resource settings.

Key points

  • Headline gains: +0.8% OOD accuracy on Amazon reviews, +1.8% OOD accuracy on MNLI, and +1.4 BLEU in-domain on IWSLT14 German-English.
  • Benchmarks:
    • Sentiment: AR-Full, AR-Clothing, Yelp, and Movies (SST2 paired with IMDb).
    • NLI: MNLI and ANLI (R1–R3).
    • Machine translation: IWSLT14 de→en, OPUS (train on medical, test on Koran, Ubuntu, subtitles and legal text), and de→rm (Allegra corpus plus Swiss press releases, tested on Convivenza blogs).
  • Ablations on reconstruction and sampling:
    • The choice of reconstruction model matters little: DistilRoBERTa (82M), RoBERTa BASE (125M) and RoBERTa LARGE (355M) give similar improvements.
    • Unrestricted sampling from the MLM outperforms top-k sampling.
  • Ablations on corruption and data size:
    • For sentiment, OOD accuracy is best at about 45% corrupted tokens, and augmented models still beat the baseline even at 95%. The NLI task, by contrast, is hurt by heavy corruption.
    • Generating more augmented sentences per example raises gains and lowers variance.
    • Benefits persist with fewer than 100 training examples.
  • Code: available at github.com/nng555/ssmba.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States136 works · 86,390 citations🎓University of TorontoToronto · Canada47 works · 50,254 citations🔬Vector InstituteToronto · Canada19 works · 4,281 citations🎓Courant Institute of Mathematical SciencesNew York · United States7 works · 8,082 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Nathan Ng, Kyunghyun Cho and Marzyeh Ghassemi. 2020. SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.97

@inproceedings{ng2020ssmba,
  title = {{SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness}},
  author = {Nathan Ng, Kyunghyun Cho and Marzyeh Ghassemi},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.97},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.97}
}

References

3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. c l Train Valid Test † AR-Clothing Men Women Baby Shoes Jianmo Ni 2019 Jianmo Ni 2019 Jianmo Ni 2019 Jianmo Ni 2019 5 5 5 5 31 40 29 41 25k 25k † 25k † 25k † 2k 2k 2k 2k 2k 2k 2k 2k AR-Full Books Clothing, Shoes & Jewelry Home and Kitchen Kindle Store Movies & TV Pet Supplies Sports & Outdoors Elect… find
  2. Nie et al. 2019 Nie et al. 2019 Nie et al. 2019 3 3 3 92 90 82 17k 46k 100k 1k 1k 1k 1k 1k 1k IWSLT IWSLT Cettolo et al. 2014 - 160k 7k 7k OPUS Medical IT Koran Law Subtitles Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 — — - 13 14 23 31 10 1.1m — — 2k 2k 2k 2k 2k 2… find
  3. AR-Full * * RNN CNN 50% 40% Unrestricted Sampling Unrestricted Sampling Preserve Label Soft Label 5 5 * * SST2 IMDb SST2 IMDb RNN CNN RNN RNN CNN CNN 60% 40% 10% 20% 60% 30% Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sam… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ng-2020-ssmba-self-supervised-manifold-based/