Elamur Research⌕
☰
home / papers / 2020 / Text Classification Using Label Names Only: A Language Model Self-Training Approach

Text Classification Using Label Names Only: A Language Model Self-Training Approach

Summary

The paper asks whether a text classifier can be trained without any annotated documents, using only each category's name plus an unlabeled corpus. The authors propose LOTClass, which uses a pre-trained BERT model in two roles: as a source of general language knowledge and as the classifier itself. Training has three stages:

  1. BERT's masked language model finds words that can substitute for each label name across many contexts, producing a category vocabulary.
  2. A new task, Masked Category Prediction (MCP), teaches the model to infer which category a masked word signals from its surrounding context. Only words judged category-indicative in that context are used as supervision.
  3. Self-training on the unlabeled corpus extends this word-level skill to whole documents.

LOTClass was tested on four benchmarks covering both topic and sentiment tasks, with one to three label words per class. It reaches roughly 90% accuracy, clearly beats earlier weakly-supervised methods, and approaches semi-supervised and supervised systems. This makes label names a cheap and underused form of supervision for settings where annotation is scarce.

Key points

  • Benchmarks: AG News, DBPedia, IMDB and Amazon. Supervision is at most 3 words per class, and usually just 1. The backbone is BERT-base-uncased with no extra resources such as back-translation systems or Wikipedia.
  • Category vocabulary construction:
    • For each label-name occurrence, the top-50 MLM predictions count as valid replacements.
    • The 100 most frequent replacements form the class vocabulary.
    • NLTK stopwords and words shared across classes are removed.
  • MCP versus simple matching: MCP addresses two problems with plain keyword matching. Keyword occurrences do not always indicate the category, and the vocabulary has limited coverage. A "BERT w. simple match" baseline gains nothing from self-training, while LOTClass steadily improves.
  • Comparison with supervised BERT: On AG News, LOTClass performs close to supervised BERT trained with about 48 labeled documents per class.
  • Limitations and code: Implicit sentiment remains hard to capture from label names alone. The authors suggest active learning and combination with semi-supervised methods as future directions. Code is released as LOTClass (github.com/yumeng5/LOTClass).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.724

@inproceedings{meng2020text,
  title = {{Text Classification Using Label Names Only: A Language Model Self-Training Approach}},
  author = {Meng, Yu and Zhang, Yunyi and Huang, Jiaxin and Xiong, Chenyan and Ji, Heng and Zhang, Chao and Han, Jiawei},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.724},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.724}
}

References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  2. Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In AAAI. find
  3. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL. find
  4. Xingyuan Chen, Yunqing Xia, Peng Jin, and John A. Carroll. 2015. Dataless text classification with descriptive lda. In AAAI. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  6. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In IJCAI. find
  7. Jiaxin Huang, Yu Meng, Fang Guo, Heng Ji, and Jiawei Han. 2020. Weakly-supervised aspect-based sentiment analysis via joint aspect-sentiment topic embedding. In EMNLP. find
  8. Nitin Jindal and Bing Liu. 2007. Review spam detection. In WWW. find
  9. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP. find our card
  10. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  11. Thomas Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR. find our card
  12. Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR. find
  13. Dong-Hyun Lee. 2013. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML. find
  14. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015. Dbpedia -a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167–195. find
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL. find our card
  16. Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. 2016. Effective document labeling with very few seed words: A topic model approach. In CIKM. find
  17. Hongmei Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao ming Wu, and Albert Y. S. Lam. 2019a. Reconstructing capsule networks for zero-shot intent classification. In EMNLP. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  19. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL. find our card
  20. Julian J. McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In RecSys ’13. find
  21. Dheeraj Mekala and Jingbo Shang. 2020. Contextualized weak supervision for text classification. In ACL. find
  22. Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020a. Discriminative topic mining via category-name guided text embedding. In WWW. find
  23. Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In CIKM. find
  24. Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2019. Weakly-supervised hierarchical text classification. In AAAI. find
  25. Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. 2020b. Hierarchical topic mining via joint spherical tree and text embedding. In KDD. find
  26. Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In ICLR. find our card
  27. Jinseok Nam, Eneldo Loza Mencı́a, and Johannes Fürnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In AAAI. find
  28. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In EMNLP. find our card
  29. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  30. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT. find
  31. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In EMNLP. find our card
  32. Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. 2017. Train once, test anywhere: Zero-shot learning for text classification. ArXiv, abs/1712.05972. — [arXiv:1712.05972](https://arxiv.org/abs/1712.05972) find
  33. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. In NAACL-HLT. find our card
  34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  35. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In EMNLP. find our card
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP. find our card
  37. Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In EMNLP. find
  38. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL. find
  39. Yangqiu Song and Dan Roth. 2014. On dataless hierarchical text classification. In AAAI. find
  40. Duyu Tang, Bing Qin, and Ting Liu. 2015a. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP. find our card
  41. Jian Tang, Meng Qu, and Qiaozhu Mei. 2015b. Pte: Predictive text embedding through large-scale heterogeneous text networks. In KDD. find
  42. Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015c. Line: Large-scale information network embedding. In WWW. find
  43. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR. find our card
  44. Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. In NAACL-HLT. find
  45. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS. find our card
  46. Yu Zhang, Yu Meng, Jiaxin Huang, Frank F. Xu, Xuan Wang, and Jiawei Han. 2020. Minimally supervised categorization of text with metadata. In SIGIR. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meng-2020-text-classification-label-names-only/