Elamur Research
☰
home / papers / 2016 / Adversarial Training Methods for Semi-Supervised Text Classification

Adversarial Training Methods for Semi-Supervised Text Classification

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1605.07725

Summary

Adversarial training (AT) and its label-free variant, virtual adversarial training (VAT), had mostly been used as regularizers for image classifiers. They work by adding tiny perturbations to many input values, which does not suit text, where words are sparse one-hot vectors. Miyato, Dai and Goodfellow adapt both techniques to text by perturbing the continuous word embeddings fed into an LSTM classifier, not the raw tokens. To stop the network from simply inflating embedding norms, they normalize embeddings using word-frequency statistics. They compute perturbations with a gradient-based linear approximation for AT and a single power-iteration step for VAT. Because VAT needs no labels, it extends naturally to semi-supervised learning. Models are first pretrained as recurrent language models (following Dai & Le, 2015). They are then evaluated on five benchmarks covering sentiment and topic classification. The authors report state-of-the-art results in both semi-supervised and fully supervised settings, with only one extra hyperparameter to tune (ε, the perturbation norm). Their analysis also shows better word embeddings and less overfitting. The work matters as an early, simple, and cheap regularizer for RNN text models that exploits plentiful unlabeled text.

Key points

  • Datasets:
    • IMDB: 25k train, 50k unlabeled.
    • Elec (Amazon electronics reviews).
    • Rotten Tomatoes: random 90/10 split repeated over 5 seeds, with about 7.9M unlabeled Amazon movie reviews added.
    • DBpedia: 14 classes, supervised only.
    • RCV1: 55 second-level topics.
  • Architecture: a unidirectional or bidirectional LSTM classifier, initialized from a pretrained recurrent language model.
  • Scope: the perturbed embeddings do not correspond to real words, so the method is presented purely as a regularizer, not as a defense against attackers.
  • Novelty: the authors describe this as the first use of AT/VAT for text or RNN models, with ε as the sole additional hyperparameter.
  • Code: a TensorFlow implementation is released in the tensorflow/models repository (research/adversarial_text).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

237 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 224 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1605.07725 · arXiv 1605.07725

@misc{miyato2016adversarial,
  title = {{Adversarial Training Methods for Semi-Supervised Text Classification}},
  author = {Miyato, Takeru and Dai, Andrew M. and Goodfellow, Ian},
  year = {2016},
  doi = {10.48550/arxiv.1605.07725},
  eprint = {1605.07725},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.07725}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heteroge-Test error rate 0.87% 0.90% 0.85% 0.79% 0.76% 3.57% 1.73% 1.41% 1.31% 1.19% 0.84% neous distribut… find our card
  2. Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. The Journal of Machine Learning Research, 7(Nov):2399– 2434, 2006. find
  3. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. Neural probabilistic language models. In Innovations in Machine Learning, pp. 137–186. Springer, 2006. find
  4. Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In AISTATS, 2005. find
  5. Ronan Collobert, Fabian Sinz, Jason Weston, and Léon Bottou. Large scale transductive svms. Journal of Machine Learning Research, 7(Aug):1687–1712, 2006. find
  6. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In NIPS, 2015. find
  7. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011. find
  8. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015. find our card
  9. Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610, 2005. find
  10. Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006. find
  11. Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In ICCV, 2009. find
  12. Thorsten Joachims. Transductive inference for text classification using support vector machines. In ICML, 1999. find
  13. Rie Johnson and Tong Zhang. Effective use of word order for text categorization with convolutional neural networks. NAACL HLT, 2015a. find
  14. Rie Johnson and Tong Zhang. Semi-supervised convolutional neural networks for text categorization via region embedding. In NIPS, 2015b. find
  15. Rie Johnson and Tong Zhang. Convolutional neural networks for text categorization: Shallow word-level vs. deep character-level. arXiv preprint arXiv:1609.00718, 2016a. — [arXiv:1609.00718](https://arxiv.org/abs/1609.00718) find
  16. Rie Johnson and Tong Zhang. Supervised and semi-supervised text categorization using LSTM for region embeddings. In ICML, 2016b. find
  17. Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014. find our card
  18. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. find
  19. Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, 2014. find our card
  20. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, et al. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195, 2015. find
  21. David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361–397, 2004. find our card
  22. Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In ICML, 2016. find
  23. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL: Human Language Technologies-Volume 1, 2011. find our card
  24. Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In ACM conference on Recommender systems, 2013. find
  25. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013. find
  27. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In ICLR, 2016. find
  28. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010. find
  29. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005. find
  30. Ben Poole, Jascha Sohl-Dickstein, and Surya Ganguli. Analyzing noise in autoencoders and deep networks. In Deep Leanring Workshop on NIPS, 2013. find
  31. J. Sietsma and R. Dow. Creating artificial neural networks that generalize. Neural Networks, 4(1), 1991. find
  32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 2014. find
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  34. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014. find our card
  35. Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL: Short Papers, 2012. find
  36. David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow (eds.), Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press. find
  37. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015. find our card
  38. Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In IJCAI, 2015. find