Elamur Research
☰
home / papers / 2018 / Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-2072.pdf

Summary

Text classifiers often overfit, and NLP has few general augmentation tools. The usual approach swaps words for WordNet synonyms, but true synonyms are rare, so only a small share of the vocabulary can be varied. Kobayashi proposes contextual augmentation. A bidirectional LSTM language model looks at the words around a position and proposes substitutes that fit there, words in a paradigmatic relation with the original rather than strict synonyms. Substitutes are sampled fresh at every training update, and a temperature parameter controls how aggressive the replacement is.

Unrestricted context-based swaps can flip a sentence's meaning, for example replacing a positive adjective with a negative one. To prevent this, the LM is extended with a label embedding so its predictions depend on the class label. The LM is pretrained on WikiText-103 and then fine-tuned on each labeled dataset.

On six classification benchmarks, with both CNN and RNN classifiers, the method beats synonym replacement, and the label-conditional variant gives the best average accuracy. The gains are modest, but the approach needs no task-specific rules or hand-built resources.

Key points

  • Evaluated on six datasets: SST5, SST2, Subj, MPQA, RT and TREC. CNN (Kim, 2014) and single-layer LSTM classifiers were used, with results averaged over eight random seeds.
  • Average CNN accuracy:
    • baseline: 77.53
    • synonym augmentation: 77.50
    • contextual augmentation: 78.02
    • contextual + label conditioning: 78.20
  • Average RNN accuracy:
    • baseline: 77.43
    • synonym augmentation: 77.40
    • contextual augmentation: 77.62
    • contextual + label conditioning: 77.83
  • In the label-conditional LM, a positive label steers predictions for "fantastic" toward words like funny and good. A negative label steers them toward words like dull and bad. Label-neutral positions, such as "actors", change little.
  • The code is written in Chainer and released at github.com/pfnet-research/contextual_augmentation. The author notes that the improvements are sometimes marginal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

201 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 189 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2072

@inproceedings{kobayashi2018contextual,
  title = {{Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations}},
  author = {Kobayashi, Sosuke},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2072},
  url = {https://doi.org/10.18653/v1/n18-2072}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, pages 1171–1179. find
  2. Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. 2017. A neural language model for dynamically representing the meanings of unknown words and entities in a discourse. In IJCNLP, pages 473–483. find
  3. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In CoNLL SIGMORPHON, pages 31–39. find
  4. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic entity representation with max-pooling improves machine reading. In Proceedings of NAACL-HLT, pages 850–855. find
  5. Koen Deschacht and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the latent words language model. In EMNLP, pages 21– 29. find
  6. Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In ACL, pages 271–276. find
  7. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In ACL, pages 334–343. find our card
  8. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097– 1105. find
  9. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In ACL, pages 567–573. find our card
  10. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING, pages 1–7. find
  11. William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better text generation via filling in the . In ICLR. find
  12. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR. find our card
  13. Hagen Fürstenau and Mirella Lapata. 2009. Semi-supervised semantic role labeling. In EACL, pages 220–228. find
  14. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. find our card
  15. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS, pages 649–657. find our card
  16. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL. find
  17. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115– 124. find
  18. Charles Bally Albert Sechehaye Saussure, Ferdinand de and Albert Riedlinger. 1916. Cours de linguistique generale. Lausanne: Payot. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL, pages 86–96. find
  20. Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. In CoNLL SIGMOR-PHON, pages 90–99. find
  21. Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorri. 1998. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation. Springer Berlin Heidelberg. find
  22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. find
  23. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  24. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In CVPR. find
  25. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on LearningSys in NIPS 28. find
  26. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using \#petpeeve tweets. In EMNLP, pages 2557–2563. find
  27. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210. find
  28. Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. 2017. Dual supervised learning. In ICML, pages 3789–3798. find
  29. Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. 2017. Variational autoencoder for semi-supervised text classification. In AAAI, pages 3358–3364. find