Elamur Research
☰
home / papers / 2021 / Noisy Channel Language Model Prompting for Few-Shot Text Classification

Noisy Channel Language Model Prompting for Few-Shot Text Classification

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2108.04106

Summary

Few-shot text classification with large causal language models usually relies on "direct" prompting: the model scores each label's verbalizer given the input. This approach is fragile. Accuracy shifts with the choice of verbalizer and random seed, and the worst runs can drop to near chance. Min et al. adapt the noisy channel idea from machine translation and reverse the scoring direction. Their model estimates how probable the input text is after a given label expression, so it must account for every input token. They apply this to settings where the LM stays frozen: zero-shot scoring, concatenated demonstrations, a new ensemble-based demonstration scheme, and prompt tuning with very few trainable parameters. On eleven text classification datasets, channel variants clearly beat their direct counterparts, mainly because they are more stable: variance is lower and worst-case accuracy is much higher. Ablations show that direct head tuning is a surprisingly strong baseline. Channel prompt tuning still wins when examples are few, labels are imbalanced, or test labels were absent from training. This makes it practical guidance for low-resource scenarios.

Key points

  • Direct scoring uses P(label | input). Channel scoring applies Bayes' rule with a uniform label prior, so only P(input | label) is computed. Length normalization follows Holtzman et al. (2021).
  • The paper compares three formulations: Direct, a calibrated Direct++ (normalized by the probability given a NULL input), and Channel. These are evaluated across zero-shot, concat-based, and the newly proposed ensemble-based demonstrations.
  • Tuned variants update under 0.01% of model parameters. The variants include prompt tuning (Lester et al., 2021), head tuning, and transformation tuning, alongside all-finetuning as a reference.
  • When one label is held out of training, every direct model fails to predict it. Channel prompt tuning can still predict it and also transfers well across datasets, especially between 2-way and 5-way sentiment tasks.
  • Head tuning remains better on tasks far from language modeling, such as TREC and Subj. Open problems include extending channel models to non-classification tasks and to masked LMs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.04106 · arXiv 2108.04106

@misc{min2021noisy,
  title = {{Noisy Channel Language Model Prompting for Few-Shot Text Classification}},
  author = {Min, Sewon and Lewis, Mike and Hajishirzi, Hannaneh and Zettlemoyer, Luke},
  year = {2021},
  doi = {10.48550/arxiv.2108.04106},
  eprint = {2108.04106},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.04106}
}

References

49 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hiba Asri, H. Mousannif, H. A. Moatassime, and Thomas Noël. 2016. Using machine learning algorithms for breast cancer risk prediction and diagnosis. In ANT/SEIT. find
  2. Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020. Self-supervised meta-learning for few-shot natural language classification tasks. In EMNLP. find
  3. Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen-tau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In NAACL. find
  4. Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics. find
  5. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? TACL. find our card
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  7. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL. find
  8. Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018. Semi-supervised sequence modeling with cross-view training. In EMNLP. find our card
  9. Xiaoan Ding and Kevin Gimpel. 2019. Latent-variable generative models for data-efficient text classification. In EMNLP. find
  10. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. find
  11. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL. find our card
  12. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In ICML. find
  13. Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In EMNLP. find our card
  14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML. find
  15. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. find
  16. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  17. Philipp Koehn, Franz J Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL-HLT. find
  18. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In NAACL-HLT. find our card
  19. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web. find
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP. find
  21. Mike Lewis and Angela Fan. 2018. Generative question answering: Learning to answer the whole question. In ICLR. find
  22. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL. find
  23. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385. find our card
  24. Robert L Logan IV, Ivana Balaževic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353. find
  25. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. find our card
  26. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172. find
  27. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In ICLR. find our card
  28. Andrew Y Ng and Michael I Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NeurIPS. find
  29. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL. find
  30. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL. find
  31. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. find our card
  32. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In NeurIPS. find our card
  33. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In NAACL-HLT. find our card
  34. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
  35. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In NeurIPS. find our card
  36. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal. find
  37. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP. find our card
  38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. find
  39. Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. In EMNLP. find
  40. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In SIGIR. find
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  42. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2020. Unsupervised data augmentation for consistency training. In NeurIPS. find our card
  43. Kenji Yamada and Kevin Knight. 2001. A syntax-based statistical translation model. In ACL. find our card
  44. Kyra Yee, Nathan Ng, Yann N Dauphin, and Michael Auli. 2019. Simple and effective noisy channel modeling for neural machine translation. In EMNLP. find
  45. Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. 2017. Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898. — [arXiv:1703.01898](https://arxiv.org/abs/1703.01898) find
  46. Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2017. The neural noisy channel. In ICLR. find
  47. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS. find our card
  48. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In ICML. find our card
  49. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [mask]: Learning vs. learning to recall. In NAACL-HLT. find our card