Elamur Research
☰
home / papers / 2021 / Learning How to Ask: Querying LMs with Mixtures of Soft Prompts

Learning How to Ask: Querying LMs with Mixtures of Soft Prompts

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2104.06599

Summary

The paper asks how to phrase cloze-style queries so that pretrained language models best reveal the relational knowledge they store. Earlier systems relied on hand-written prompts (LAMA), mined or paraphrased prompts (LPAQA), or discrete gradient-guided word search (AutoPrompt). Qin and Eisner instead treat prompts as "soft" continuous vectors that need not match real word embeddings. They tune these vectors by gradient descent while the LM itself stays frozen. They also learn mixture weights that ensemble several prompts per relation, and optionally perturb the prompt's hidden states at every Transformer layer. Experiments cover T-REx (original and extended), Google-RE and ConceptNet, querying BERT-base, BERT-large, RoBERTa and BART. The tuned prompts beat all earlier prompting methods by wide margins on held-out pairs. This holds even when prompts start from random Gaussian vectors rather than human-designed wording. The work matters because it shows that earlier probing underestimated how much factual knowledge LMs contain, and that cheap prompt optimization can elicit it without fine-tuning the model.

Key points

  • On T-REx original with BERT-base, P@1 rises from 31.1 (LAMA) and 43.3 (AutoPrompt) to 50.7 with soft prompts initialized from mined prompts. BERT-large reaches 51.9 with random initialization.
  • On ConceptNet, LAMA scores only 0.1 P@1. Randomly initialized soft prompts on BERT-large reach 14.5 P@1 and 38.6 P@10. On Google-RE with BERT-large, P@1 improves from 9.7 to 12.9.
  • In the ablation (BERT-large, T-REx original), tuning only the mixture weights moves P@1 from 39.4 to 40.0. Tuning the token vectors gives most of the gain (50.7), and tuning both reaches 51.0.
  • Perturbing all layers usually gives small extra gains but clearly hurts RoBERTa. Data-dependent mixture weights add little, because the learned temperature becomes high.
  • The researchers released their code and data at github.com/hiaoxui/soft-prompts. An appendix tests a harder split in which train and test share no answer (y) values, and soft prompts still help somewhat there.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

238 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 218 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.410 · arXiv 2104.06599

@misc{qin2021learning,
  title = {{Learning How to Ask: Querying LMs with Mixtures of Soft Prompts}},
  author = {Qin, Guanghui and Eisner, Jason},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.410},
  eprint = {2104.06599},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.410}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Association for Computational Linguistics (ACL), pages 238–247. find our card
  2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence (AAAI). find our card
  3. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Association for the Advancement of Artificial Intelligence (AAAI), volume 34, pages 7456–7463. find
  4. David Broughton. 1995. The assumptions and theory of public opinion polling. In Public Opinion Polling and Politics in Britain, pages 15–33. Springer. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  6. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL). find
  7. M. C. Donaldson. 1978. Children’s Minds. W. W. Norton. find
  8. Max Eichler, Gözde Gül Şahin, and Iryna Gurevych. 2019. LINSPECTOR WEB: A multilingual probing suite for word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I… find
  9. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Simperl, and Frederique Laforest. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Language Resources and Evaluation Conference (LREC), page 5. find
  10. M. A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In International Conference on Computational Linguistics (COL-ING). find
  11. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics (TACL). find our card
  12. D. P. Kingma and J. L. Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), pages 1– 15. find
  13. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In North American Association for Computational Linguistics and Human Language Technology (NAACL-HLT), pages 260–270. find
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACL). find our card
  15. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv preprint arXiv:2005.11401. find
  16. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  17. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv preprint arXiv:2103.10385. find our card
  18. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. S. Zettlemoyer, and V. Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. find
  19. Dave Orr. 2013. 50,000 lessons on how to read: A relation extraction corpus. https://github. com/google-research-datasets/ relation-extraction-corpus. find
  20. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. S. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). find
  21. F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. 2019. Language models as knowledge bases? In Empirical Methods in Natural Language Processing (EMNLP). find our card
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  23. Delip Rao, Paul McNamee, and Mark Dredze. 2013. Entity linking: Finding extracted entities in a knowledge base. In Multi-Source, Multilingual Information Extraction and Summarization, pages 93–115. Springer. find
  24. E. Riloff and R. Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Association for the Advancement of Artificial Intelligence (AAAI), pages 474–479. find
  25. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics (TACL). find our card
  26. Kenneth Rose. 1998. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 80:2210–2239. find
  27. Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. Accepted to EACL 2021. find our card
  28. Timo Schick and Hinrich Schütze. 2020b. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. find our card
  29. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  30. Push Singh, Thomas Lin, Erik T. Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open Mind Common Sense: Knowledge acquisition from the general public. In On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE, volume 2519, pages 1223–1237. Springer. find
  31. Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 1784–1789. find
  32. R. Speer, J. Chin, and C. Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Association for the Advancement of Artificial Intelligence (AAAI). find
  33. Mihai Surdeanu and Heng Ji. 2014. Overview of the English slot filling track at the TAC2014 knowledge base population evaluation. In Proceedings of the TAC-KBP 2014 Workshop. find
  34. Alon Talmor, Yanal Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics – On what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743–758. find our card