Elamur Research⌕
☰
home / papers / 2020 / Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

Paper · 2020 · Timo Schick, Helmut Schmid and Hinrich Schütze

Summary

Pattern-Exploiting Training (PET) tackles few-shot text classification by rewriting each input as a cloze-style prompt for a pretrained masked language model. The label is then inferred from the word the model predicts at the mask. This depends on a verbalizer, a hand-picked word for each label, and choosing one requires both task expertise and intuition about which words the model understands well. The authors propose PETAL, which derives verbalizers automatically from a small labeled set. Searching over all word-to-label assignments directly is infeasible, so they split the task into one-vs-rest subproblems and reweight examples to offset the resulting class imbalance. Candidate tokens are scored with a likelihood-ratio loss rather than plain cross entropy, which would otherwise reward words that are simply probable everywhere. Candidates are prefiltered by frequency and fit to the pattern, and several words can jointly represent a label (multi-verbalizers). With only 50 training examples on Yelp, AG's News, Yahoo Questions and MNLI, PETAL outperforms supervised fine-tuning and the earlier automatic verbalizer search (AVS). It comes close to hand-crafted verbalizers, reducing the expertise needed to apply PET.

Key points

  • Average accuracy across the four tasks with |T| = 50 labeled examples:
    • supervised baseline: 56.3
    • PET + random verbalizers: 57.2
    • PET + AVS: 62.8
    • PETAL (joint): 65.9
    • PETAL (sep): 66.4
    • PET with manual verbalizers: 69.1
  • The largest gain over AVS is on MNLI: 62.4 for PETAL (sep) versus 52.6 for AVS, close to the manual upper bound of 63.9. PETAL beats AVS on 3 of the 4 tasks; on AG's News the two are nearly tied.
  • Candidate vocabulary is built in two steps. First, take the 10,000 most frequent real-word tokens in the task's unlabeled data. Then keep, for each label, the 1,000 tokens that best fit the positive examples. This deliberately favors frequent words, which language models understand better.
  • The only new hyperparameter is the number of verbalizations per label (n_v). Main experiments use n_v = 10. Results stay stable for values from 1 to 100, and n_v = 3 gives the best overall score.
  • The implementation is released in the PET repository (github.com/timoschick/pet). Experiments also use 10,000·k unlabeled examples per task, but these serve only PET's distillation step, not the verbalizer search.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany44 works · 55,193 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Timo Schick, Helmut Schmid and Hinrich Schütze. 2020. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.488

@inproceedings{schick2020automatically,
  title = {{Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification}},
  author = {Timo Schick, Helmut Schmid and Hinrich Schütze},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.488},
  url = {https://doi.org/10.18653/v1/2020.coling-main.488}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM Conference on Digital Libraries, DL ’00, page 85–94, New York, NY, USA. Association for Computing Machinery. find
  2. David S. Batista, Bruno Martins, and Mário J. Silva. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 499–504, Lisbon, Portugal, September. Association for C… find
  3. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. find
  4. Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In Paolo Atzeni, Alberto Mendelzon, and Giansalvatore Mecca, editors, The World Wide Web and Databases, pages 172–183, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  6. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1… our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  8. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48, Jan. our card
  9. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2019. How can we know what language models know? Computing Research Repository, arXiv:1911.12543. our card
  10. Yoonkyung Lee, Yi Lin, and Grace Wahba. 2001. Multicategory support vector machines. Technical report, Department of Statistics, University of Madison, Wisconsin. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692. find
  12. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 193–202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. find
  13. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La… our card
  14. Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. Computing Research Repository, arXiv:1912.10165. — [arXiv:1912.10165](https://arxiv.org/abs/1912.10165) find
  15. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find
  17. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Computing Research Repository, arXiv:1910.10683. our card
  18. Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze questions for few shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676. our card
  19. Timo Schick and Hinrich Schütze. 2020b. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. find
  20. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. Computing Research Repository, arXiv:2004.05483. find
  21. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. oLMpics – on what language model pre-training captures. Computing Research Repository, arXiv:1912.13283. our card
  22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  23. Hengshuai Yao, Dong-lai Zhu, Bei Jiang, and Peng Yu. 2020. Negative log likelihood ratio loss for deep neural network classification. In Kohei Arai, Rahul Bhatia, and Supriya Kapoor, editors, Proceedings of the Future Technologies Conference (FTC) 2019, pages 276–282, Cham. Springer International Pu… find
  24. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 649–657. Curran Associates, Inc. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schick-2020-automatically-identifying-words-that-can/