Elamur Research⌕
☰
home / papers / 2020 / MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Paper · 2020 · Jiaao Chen, Zichao Yang and Diyi Yang

Summary

Text classifiers trained on very few labeled examples tend to overfit. Earlier semi-supervised approaches also treat labeled and unlabeled data separately, so supervision does not flow between them. Chen, Yang and Yang adapt Mixup, an image augmentation technique, to text. Their method, TMix, encodes two sentences with 12-layer BERT-base and linearly blends their hidden states at a chosen layer. The blended state then passes through the remaining layers, and the model is trained against a correspondingly blended label. Because the weight is continuous, this yields an effectively unlimited supply of synthetic training points. MixText builds on TMix. It first guesses sharpened, low-entropy labels for unlabeled texts by averaging predictions over the originals and their back-translations. It then interpolates labeled, unlabeled and augmented examples together, adding entropy minimization and consistency regularization. Evaluated across four benchmarks (IMDB, Yahoo! Answer, DBpedia and AG News), MixText outperforms BERT fine-tuning, VAMPIRE and UDA, and its advantage is largest when just 10 labels per class are provided. The work matters for low-resource settings where annotation is expensive but raw text is plentiful. The code is publicly available.

Key points

  • With 10 labeled examples per class, supervised TMix alone raised AG News accuracy from 69.5% (BERT) to 74.1%. On IMDB, MixText reached 78.7%, against 67.5% for BERT and 78.2% for UDA.
  • Experiments used 5000 unlabeled texts per class. Augmentation was back-translation through German and Russian (K = 2), and the mixing weight λ was drawn from a Beta(α, α) distribution and set to max(λ, 1−λ).
  • The choice of mixing layer matters. On AG News with 10 labels, mixing at layers {7, 9, 12} scored best (74.1%), while low layers {0, 1, 2} gave no gain (69.3%).
  • In the reported ablation, removing TMix dropped accuracy from 67.6% to 63.5%, and removing unlabeled data dropped it to 58.6%. Removing all components dropped it to 56.2%.
  • Development-set loss curves showed BERT beginning to overfit after about 10 epochs, while TMix and MixText stayed lower and more stable. Code is released at github.com/GT-SALT/MixText.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Jiaao Chen, Zichao Yang and Diyi Yang. 2020. MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.194

@inproceedings{chen2020mixtext,
  title = {{MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification}},
  author = {Jiaao Chen, Zichao Yang and Diyi Yang},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.194},
  url = {https://doi.org/10.18653/v1/2020.acl-main.194}
}

References

47 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p…
  2. David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. CoRR, abs/1905.02249. — [arXiv:1905.02249](https://arxiv.org/abs/1905.02249) find
  3. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 10–21.
  4. Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence -Volume 2, AAAI’08, pages 830–835. AAAI Press. \#\# Conclusion To alleviate the dependencies of sup… — [arXiv:1109.2047](https://arxiv.org/abs/1109.2047) find
  5. Jiaao Chen, Jianshu Chen, and Zhou Yu. 2019. Incorporating structured commonsense knowledge in story completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Sympos… find
  6. Jiaao Chen, Yuwei Wu, and Diyi Yang. 2020. Semi-supervised Models via Data Augmentation for Classifying Interactive Affective Responses. In Workshop On Affective Content Analysis, The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020. find
  7. Mingda Chen, Qingming Tang, Karen Livescu, and Kevin Gimpel. 2018. Variational sequential labelers for semi-supervised learning. In Proc. of EMNLP. find
  8. Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018. Semi-supervised sequence modeling with cross-view training. arXiv preprint arXiv:1809.08370.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… find
  11. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. CoRR, abs/1808.09381. find
  12. Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS’04, pages 529–536, Cambridge, MA, USA. MIT Press. find
  13. Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. CoRR, abs/1906.02242. find
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  15. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  16. Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019. find
  17. Submodular Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. CoRR, abs/1610.02242. — [arXiv:1610.02242](https://arxiv.org/abs/1610.02242) find
  18. Pooja Lakshmi Narayan, Ajay Nagesh, and Mihai Surdeanu. 2019. Exploration of noise strategies in semi-supervised named entity classification. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 186– 191, Minneapolis, Minnesota. Association for Comp… find
  19. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. find
  20. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature Cell Biology, 521(7553):436–444. find
  21. Dong-Hyun Lee. 2013. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. find
  22. ICML 2013 Workshop : Challenges in Representation Learning (WREPL). find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  24. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 142–1…
  25. Pablo N. Mendes, Max Jakob, and Christian Bizer. 2012. Dbpedia for nlp: A multilingual cross-domain knowledge base. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. find
  26. Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, pages 983–992, New York, NY, USA. ACM. find
  27. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations.
  28. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(8):1979–1993. find
  29. Ajay Nagesh and Mihai Surdeanu. 2018. An exploration of three lightly-supervised representation learning approaches for named entity classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2312–2324. find
  30. Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. CoRR, abs/1712.04621. — [arXiv:1712.04621](https://arxiv.org/abs/1712.04621) find
  31. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. CoRR, abs/1511.06709. find
  34. Antti Tarvainen and Harri Valpola. 2017. Weight-averaged consistency targets improve semi-supervised deep learning results. CoRR, abs/1703.01780. — [arXiv:1703.01780](https://arxiv.org/abs/1703.01780) find
  35. Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019a. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine… find
  36. Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019b. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3635–3641. International Joint Conferences… find
  37. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848.
  38. Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019a. Let’s make your request more persuasive: Modeling persuasive strategies via semi-supervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp… find
  39. Diyi Yang, Miaomiao Wen, and Carolyn Rose. 2015. Weakly supervised role identification in teamwork interactions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers… find
  40. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
  41. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. CoRR, abs/1702.08139. — [arXiv:1702.08139](https://arxiv.org/abs/1702.08139) find
  42. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 14…
  43. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. CoRR, abs/1905.04899. — [arXiv:1905.04899](https://arxiv.org/abs/1905.04899) find
  44. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. CoRR, abs/1710.09412.
  45. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. CoRR, abs/1509.01626.
  46. Vikas Verma, Meng Qu, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. 2019c. Graphmix: Regularized training of graph neural networks for semi-supervised learning. ArXiv, abs/1909.11715. — [arXiv:1909.11715](https://arxiv.org/abs/1909.11715) find
  47. Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. CoRR, abs/1901.11196.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2020-mixtext-linguistically-informed-interpolation-hidden/