Elamur Research
☰
home / papers / 2019 / Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1404.pdf

Summary

Zero-shot text classification (0SHOT-TC) has been studied mostly as topic categorization, with inconsistent datasets and evaluation protocols that make results hard to compare. Yin, Hay and Roth build a benchmark covering three label aspects: topic (Yahoo), emotion (a merged collection from several domains) and situation frames (event-level needs such as food or shelter in disasters). Each dataset has fixed train/dev/test splits and a fixed division into seen and unseen classes. The benchmark has two protocols. In label-partially-unseen evaluation, a model trains on some classes and predicts over all of them. In label-fully-unseen evaluation, no task-specific labeled data is available at all. To handle both, the authors cast classification as textual entailment. The input text is the premise, and each candidate label becomes a hypothesis such as "the text is about sports". Models pretrained on MNLI, FEVER and RTE stayed fairly robust across aspects without seeing any classification data. An ensemble of them performed best. A Wikipedia-trained baseline helped only for topics. The released benchmark gives future 0SHOT-TC work a common reference point.

Key points

  • Yahoo topic data: the original corpus has 10 classes and 1.4M/60k train/test examples. It was reorganized into dev (6k per label) and test (10k per label) sets. Two non-overlapping training sets, Train-v0 and Train-v1, each contain 5 classes with 130k instances per class.
  • Emotion data: it combines tweets, events, fairytales and artificial sentences, with 16,000 test and 7,700 dev instances. Labels include joy, anger, shame, guilt, love and none.
  • Wikipedia-based BERT baseline: a binary classifier trained on 100K articles. Each article is paired with its own categories plus three randomly sampled negative categories. It was strong on topic detection but weak on emotion and situation detection.
  • Ranking of entailment models: in the label-fully-unseen setting, RTE outperformed FEVER, which outperformed MNLI. After fine-tuning in the label-partially-unseen setting, MNLI-based models came out on top.
  • Hypothesis construction: hypotheses were built from the label word, its WordNet definition, or both. Definitions alone performed poorly. For situation detection, abstract classes such as evacuation, infrastructure and regime change remained difficult, while water, shelter and food were recognized well.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

214 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 207 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1404 · arXiv 1909.00161

@inproceedings{yin2019benchmarking,
  title = {{Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach}},
  author = {Yin, Wenpeng and Hay, Jamaal and Roth, Dan},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1404},
  eprint = {1909.00161},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1404}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Ana Maria Bostan and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In COLING, pages 2104–2119. find
  2. Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In AAAI, pages 830–835. find
  3. Xingyuan Chen, Yunqing Xia, Peng Jin, and John A. Carroll. 2015. Dataless text classification with descriptive LDA. In AAAI, pages 2224–2231. find
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, pages 177–… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  6. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In IJCAI, pages 1606–1611. find
  7. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In CoNLL, pages 333–342. find
  8. Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. 2016a. Effective document labeling with very few seed words: A topic model approach. In CIKM, pages 85–94. find
  9. Yuezhang Li, Ronghuo Zheng, Tian Tian, Zhiting Hu, Rahul Iyer, and Katia P. Sycara. 2016b. Joint embedding of hierarchical categories and entities for concept categorization and dataless classification. In COLING, pages 2678–2688. find
  10. Stephen Mayhew, Tatiana Tsygankova, Francesca Marini, Zihan Wang, Jane Lee, Xiaodong Yu, Xingyu Fu, Weijia Shi, Zian Zhao, Wenpeng Yin, Karthikeyan K, Jamaal Hay, Michael Shur, Jennifer Sheffield, and Dan Roth. 2019. University of pennsylvania lorehlt 2019 submission. find
  11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  12. Aldrian Obaja Muis, Naoki Otani, Nidhi Vyas, Ruochen Xu, Yiming Yang, Teruko Mitamura, and Eduard Hovy. 2018. Low-resource cross-lingual event type detection via distant supervision with minimal effort. In COLING, pages 70–82. find
  13. Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 72–78. find
  14. Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. 2017. Train once, test anywhere: Zero-shot learning for text classification. CoRR, abs/1712.05972. — [arXiv:1712.05972](https://arxiv.org/abs/1712.05972) find
  15. Marek Rei and Anders Søgaard. 2018. Zero-shot sequence labeling: Transferring knowledge from sentences to tokens. In NAACL-HLT, pages 293–302. find
  16. Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In EMNLP, pages 3132–3142. find
  17. Stephanie M. Strassel, Ann Bies, and Jennifer Tracey. 2017. Situational awareness for low resource languages: the LORELEI situation frame annotation task. In SMERP@ECIR, pages 32–41. find
  18. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT, pages 809–819. find our card
  19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. find our card
  20. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, pages 1112–1122. find
  21. Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S. Yu. 2018. Zero-shot user intent detection via capsule neural networks. In EMNLP, pages 3090–3099. find
  22. Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. 2017. Generative and discriminative text classification with recurrent neural networks. CoRR, abs/1703.01898. — [arXiv:1703.01898](https://arxiv.org/abs/1703.01898) find
  23. Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. In NAACL. find
  24. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS, pages 649–657. find our card
  25. Ben Zhou, Daniel Khashabi, Chen-Tse Tsai, and Dan Roth. 2018. Zero-shot open entity typing as type-compatible grounding. In EMNLP, pages 2065– 2076. find
  26. Prateek Veeranna Sappadla, Jinseok Nam, Eneldo Loza Menc’ia, and Johannes Fürnkranz. 2016. Using semantic similarity for multi-label zero-shot classification of text documents. In 24th European Symposium on Artificial Neural Networks, pages 423–428. find
  27. Yangqiu Song and Dan Roth. 2014. On dataless hierarchical text classification. In AAAI, pages 1579– 1585. find
  28. Yangqiu Song, Shyam Upadhyay, Haoruo Peng, and Dan Roth. 2016. Cross-lingual dataless classification for many languages. In IJCAI, pages 2901– 2907. find
  29. Shashank Srivastava, Igor Labutov, and Tom M. Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In ACL, pages 306–316. find