Elamur Research⌕
☰
home / papers / 2023 / Unleashing the potential of prompt engineering for large language models

Unleashing the potential of prompt engineering for large language models

Paper · 2023 · Banghao Chen, Zhaofeng Zhang, Nicolas Langrene et al.

Summary

This survey asks how users can get more reliable, accurate and relevant output from large language models by shaping their inputs. It does not introduce new data or experiments. Instead, it organizes existing prompting research into a structured overview. It first traces how input design has mattered across AI history, from rule-based systems in the 1950s to the transformer era, which is the review's main focus. It then covers:

  • basic tactics, such as precise instructions, role-prompting and iterative refinement, illustrated with GPT-4 outputs;
  • advanced strategies, such as chain-of-thought, self-consistency and generated knowledge;
  • learnable prompting for vision-language models.

The review also covers how prompt methods are evaluated, where they are applied, and how adversarial prompts threaten LLM security. Its main conclusion is that careful prompt design substantially improves model behaviour and can reduce hallucinations. It also argues that defences such as adversarial training are needed. The authors see better understanding of model internals and AI agents as the next frontier. The paper is a practical map for anyone building LLM-based NLP tools.

Key points

  • All illustrative examples were generated with GPT-4. The paper notes that GPT-4 is estimated to be far larger than GPT-3's 175 billion parameters.
  • VLM prompting methods reviewed include CoOp (Context Optimization), CoCoOp (Conditional Context Optimization) and MaPLe (Multimodal Prompt Learning), with CLIP and ALIGN as reference models.
  • Prompt methods are assessed through both subjective (human) and objective (metric-based) evaluation. Applications discussed include education, content creation, programming and reasoning.
  • The authors describe agent evolution in five stages: models, prompt templates, chains, agents and multi-agents. They define an agent as a large model combined with memory, planning and tool use.
  • The paper was published in Patterns 6(6), 101,260 (2025). It was supported by the Natural Science Foundation of China (grant 12,271,047).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🎓University of MichiganAnn Arbor · United States36 works · 12,181 citations🎓Beijing Normal UniversityBeijing · China6 works · 3,107 citations

Also on the byline: Beijing Normal-Hong Kong Baptist University.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Banghao Chen, Zhaofeng Zhang, Nicolas Langrene and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering for large language models. arXiv:2310.14735 · Patterns 6(6) 101,260 (2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1016/j.patter.2025.101260 · arXiv 2310.14735

@misc{chen2023unleashing,
  title = {{Unleashing the potential of prompt engineering for large language models}},
  author = {Banghao Chen, Zhaofeng Zhang, Nicolas Langrene and Shengxin Zhu},
  year = {2023},
  doi = {10.1016/j.patter.2025.101260},
  eprint = {2310.14735},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1016/j.patter.2025.101260}
}

References

305 references, 106 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17; 2017. p. 6000–6010.
  2. Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency; 2021. p. 610–623. find
  3. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS’20; 2020. p. 1877–1901.
  4. OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al. GPT-4 technical report; 2024. ArXiv:2303.08774.
  5. Team G, Anil R, Borgeaud S, Alayrac JB, Yu J, Soricut R, et al. Gemini: a family of highly capable multimodal models; 2024. ArXiv:2312.11805.
  6. Google.: Google Gemini: next-generation model. Available from: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/ \#sundar-note. ↗
  7. Hulbert D.: Tree of knowledge: ToK aka Tree of Knowledge dataset for large language models LLM. Accessed: 2023-8-15. https://github.com/dave1010/ tree-of-thought-prompting. ↗
  8. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Anthropic; 2024. https://paperswithcode.com/paper/ the-claude-3-model-family-opus-sonnet-haiku. ↗
  9. Anthropic.: Claude 3 model. Available from: https://www.anthropic.com/news/ claude-3-family. ↗
  10. Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: open foundation and fine-tuned chat models; 2023. ArXiv:2307.09288. find
  11. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, et al. The Llama 3 herd of models; 2024. ArXiv2407.21783. find
  12. Sarkhel R, Huang B, Lockard C, Shiralkar P. Self-training for label-efficient information extraction from semi-structured web-pages. Proceedings of the VLDB Endowment. 2023;16(11):3098–3110. find
  13. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, et al. Zero-shot text-to-image generation; 2021. ArXiv:2102.12092.
  14. Marcus G, Davis E, Aaronson S. A very preliminary analysis of DALL-E 2; 2022. ArXiv:2204.13807. — [arXiv:2204.13807](https://arxiv.org/abs/2204.13807) find
  15. OpenAI.: DALL·E: creating images from text. dall-e/. https://openai.com/index/ ↗
  16. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning; 2021. p. 8748–8763.
  17. Li Y, Liang F, Zhao L, Cui Y, Ouyang W, Shao J, et al. Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. In: Tenth International Conference on Learning Representations; 2022. . find
  18. OpenAI.: Hello GPT-4o. Accessed: 2024-08-04. https://openai.com/index/ hello-gpt-4o/. ↗
  19. Moore O.: Announcing GPT-4o in the API! Accessed: 2024-05-22. Available from: https://community.openai.com/t/announcing-gpt-4o-in-the-api/744700. ↗
  20. Kaddour J, Harris J, Mozes M, Bradley H, Raileanu R, McHardy R. Challenges and applications of large language models; 2023. ArXiv:2307.10169.
  21. Lu Y, Bartolo M, Moore A, Riedel S, Stenetorp P. Fantastically ordered prompts and where to find them: overcoming few-shot prompt order sensitivity. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics; 2022. p. 8086–8098.
  22. Webson A, Pavlick E. Do prompt-based models really understand the meaning of their prompts? In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2022. p. 2300–2344.
  23. Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems. vol. 35; 2022. p. 24824–24837.
  24. Wang X, Wei J, Schuurmans D, Le QV, Chi EH, Narang S, et al. Self-consistency improves chain of thought reasoning in language models. In: Eleventh International Conference on Learning Representations; 2023. .
  25. Shanahan M, McDonell K, Reynolds L. Role-play with large language models; 2023. ArXiv:2305.16367.
  26. Maynez J, Narayan S, Bohnet B, McDonald R. On faithfulness and factuality in abstractive summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020. p. 1906–1919. find
  27. Bubeck S, Chandrasekaran V, Eldan R, Gehrke J, Horvitz E, Kamar E, et al. Sparks of artificial general intelligence: early experiments with GPT-4; 2023. ArXiv:2303.12712.
  28. Yong G, Jeon K, Gil D, Lee G. Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model. Computer-Aided Civil and Infrastructure Engineering. 2022;38(11):1536–1554. find
  29. Wang J, Liu Z, Zhao L, Wu Z, Ma C, Yu S, et al. Review of large vision models and visual prompt engineering. Meta-Radiology. 2023;1(3):100047. find
  30. Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. Advances in neural information processing systems. 2017;30. find
  31. Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, et al. Learning to summarize with human feedback. Advances in Neural Information Processing Systems. 2020;33:3008–3021. find
  32. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I, et al.: Language models are unsupervised multitask learners. Assessed: 2019-02-07. https://cdn.openai.com/better-language-models/language models are unsupervised multitask learners.pdf. ↗
  33. Radford A, Narasimhan K, Salimans T, Sutskever I, et al. Improving language understanding by generative pre-training; 2018. Https://openai.com/research/language-unsupervised. find
  34. Holtzman A, Buys J, Du L, Forbes M, Choi Y. The curious case of neural text degeneration. In: Ninth International Conference on Learning Representations; 2020. . find
  35. Welleck S, Kulikov I, Roller S, Dinan E, Cho K, Weston J. Neural text generation with unlikelihood training; 2019. ArXiv:1908.04319.
  36. Xu X, Tao C, Shen T, Xu C, Xu H, Long G, et al. Re-reading improves reasoning in language models; 2023. ArXiv:2309.06275. — [arXiv:2309.06275](https://arxiv.org/abs/2309.06275) find
  37. YanSong S, JingLi Tencent A. Joint learning embeddings for Chinese words and their components via ladder structured networks. In: Proceedings of the Twenty-Seventh International Joint Conference on Artifificial Intelligence (IJCAI-18); 2018. p. 4375–4381. find
  38. Luo L, Ao X, Song Y, Li J, Yang X, He Q, et al. Unsupervised neural aspect extraction with sememes. In: IJCAI; 2019. p. 5123–5129. find
  39. Yang M, Qu Q, Tu W, Shen Y, Zhao Z, Chen X. Exploring human-like reading strategy for abstractive text summarization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33; 2019. p. 7362–7369. find
  40. Zhang Z, Gao J, Dhaliwal RS, Jia-Jun Li T. VISAR: a human-AI argumentative writing assistant with visual programming and rapid draft prototyping; 2023. ArXiv:2304.07810. — [arXiv:2304.07810](https://arxiv.org/abs/2304.07810) find
  41. Van Buren D. Guided scenarios with simulated expert personae: a remarkable strategy to perform cognitive work; 2023. ArXiv:2306.03104. — [arXiv:2306.03104](https://arxiv.org/abs/2306.03104) find
  42. OpenAI.: Tactic: use delimiters to clearly indicate distinct parts of the input. Accessed: 2023-09-01. https://platform.openai.com/docs/guides/gpt-best-practices/ tactic-use-delimiters-to-clearly-indicate-distinct-parts-of-the-input. ↗
  43. Logan IV R, Balažević I, Wallace E, Petroni F, Singh S, Riedel S. Cutting down on prompts and parameters: simple few-shot learning with language models. In: Findings of the Association for Computational Linguistics: ACL 2022; 2022. p. 2824–2835.
  44. Shyr C, Hu Y, Harris PA, Xu H. Identifying and extracting rare disease phenotypes with large language models; 2023. ArXiv:2306.12656. — [arXiv:2306.12656](https://arxiv.org/abs/2306.12656) find
  45. Reynolds L, McDonell K. Prompt programming for large language models: beyond the few-shot paradigm. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems; 2021. p. 1–7.
  46. Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput Surv. 2023 jan;55(9).
  47. Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems. 2022;35:22199–22213.
  48. Liu J, Gardner M, Cohen SB, Lapata M. Multi-step inference for reasoning over paragraphs; 2020. ArXiv:2004.02995. find
  49. Ackley DH, Hinton GE, Sejnowski TJ. A learning algorithm for Boltzmann machines. Cognitive Science. 1985;9(1):147–169. find
  50. Ficler J, Goldberg Y. Controlling linguistic style aspects in neural language generation. In: Proceedings of the Workshop on Stylistic Variation; 2017. p. 94–104.
  51. Xu C, Guo D, Duan N, McAuley J. RIGA at SemEval-2023 Task 2: NER enhanced with GPT-3. In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023); 2023. p. 331–339. find
  52. Liesenfeld A, Dingemanse M. Rethinking open source generative AI: open washing and the EU AI Act. In: The 2024 ACM Conference on Fairness, Accountability, and Transparency; 2024. p. 1774–1787. find
  53. Wu S, Shen EM, Badrinath C, Ma J, Lakkaraju H. Analyzing chain-of-thought prompting in large language models via gradient-based feature attributions; 2023. ArXiv:2307.13339. find
  54. Zhang Z, Zhang A, Li M, Smola A. Automatic chain of thought prompting in Large language models. In: Eleventh International Conference on Learning Representations; 2023. .
  55. Lewkowycz A, Andreassen A, Dohan D, Dyer E, Michalewski H, Ramasesh V, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems. 2022;35:3843–3857.
  56. Zhou H, Nova A, Larochelle H, Courville A, Neyshabur B, Sedghi H. Teaching algorithmic reasoning via in-context learning; 2022. ArXiv:2211.09066. find
  57. Lee N, Sreenivasan K, Lee JD, Lee K, Papailiopoulos D. Teaching arithmetic to small transformers; 2023. ArXiv:2307.03381. find
  58. Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Large language models perform diagnostic reasoning. In: Eleventh International Conference on Learning Representations; 2022. . find
  59. Zhang H, Parkes DC. Chain-of-thought reasoning is a policy improvement operator; 2023. ArXiv:2309.08589. find
  60. Cobbe K, Kosaraju V, Bavarian M, Chen M, Jun H, Kaiser L, et al. Training verifiers to solve math word problems; 2021. ArXiv:2110.14168.
  61. Huang S, Dong L, Wang W, Hao Y, Singhal S, Ma S, et al. Language is not all you need: aligning perception with language models; 2023. ArXiv:2302.14045.
  62. Shum K, Diao S, Zhang T. Automatic prompt augmentation and selection with chain-of-thought from labeled data. In: Bouamor H, Pino J, Bali K, editors. Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics; 2023. p. 12113–12139.
  63. Del M, Fishel M. True detective: a deep abductive reasoning benchmark undoable for GPT-3 and challenging for GPT-4. In: Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023); 2023. p. 314–322. find
  64. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. 2022;35:27730–27744.
  65. Saparov A, He H. Language models are greedy reasoners: a systematic formal analysis of chain-of-thought; 2022. ArXiv:2210.01240.
  66. Tafjord O, Dalvi B, Clark P. ProofWriter: generating implications, proofs, and abductive statements over natural language. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021; 2021. p. 3621–3634.
  67. Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, et al. Palm: scaling language modeling with pathways; 2022. ArXiv:2204.02311.
  68. Fan A, Lewis M, Dauphin Y. Hierarchical neural story generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2018. p. 889–898. find
  69. Holtzman A, Buys J, Forbes M, Bosselut A, Golub D, Choi Y. Learning to write with cooperative discriminators. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2018. p. 1638–1649.
  70. Huang J, Gu SS, Hou L, Wu Y, Wang X, Yu H, et al. Large language models can self-improve; 2022. ArXiv:2210.11610.
  71. Shum K, Diao S, Zhang T. Automatic prompt augmentation and selection with chain-of-thought from labeled data; 2023. ArXiv:2302.12822.
  72. Khalifa M, Logeswaran L, Lee M, Lee H, Wang L. Discriminator-guided multi-step reasoning with language models; 2023. ArXiv:2305.14934. — [arXiv:2305.14934](https://arxiv.org/abs/2305.14934) find
  73. Liu J, Liu A, Lu X, Welleck S, West P, Le Bras R, et al. Generated knowledge prompting for commonsense reasoning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2022. p. 3154–3169. find
  74. Zhou D, Schärli N, Hou L, Wei J, Scales N, Wang X, et al. Least-to-most prompting enables complex reasoning in Large language models. In: Eleventh International Conference on Learning Representations; 2023. .
  75. Gao L, Madaan A, Zhou S, Alon U, Liu P, Yang Y, et al. Pal: program-aided language models. In: International Conference on Machine Learning. PMLR; 2023. p. 10764–10799. find
  76. Patel A, Bhattamishra S, Goyal N. Are NLP Models really able to Solve Simple Math Word Problems? In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics; 2021…
  77. Yao S, Yu D, Zhao J, Shafran I, Griffiths TL, Cao Y, et al. Tree of thoughts: deliberate problem solving with large language models; 2023. ArXiv:2305.10601. find
  78. Long J. Large language model guided tree-of-thought; 2023. ArXiv:2305.08291. find
  79. Besta M, Blach N, Kubicek A, Gerstenberger R, Gianinazzi L, Gajda J, et al. Graph of thoughts: solving elaborate problems with large language models; 2023. ArXiv:2308.09687. find
  80. Wang L, Ma C, Feng X, Zhang Z, Yang H, Zhang J, et al. A survey on large language model based autonomous agents; 2023. ArXiv:2308.11432.
  81. Khot T, Trivedi H, Finlayson M, Fu Y, Richardson K, Clark P, et al. Decomposed prompting: a modular approach for solving complex tasks; 2023. ArXiv:2210.02406. find
  82. Diao S, Wang P, Lin Y, Zhang T. Active prompting with chain-of-thought for large language models; 2024. ArXiv:2302.12246. find
  83. Li XL, Liang P. Prefix-tuning: optimizing continuous prompts for generation. In: Zong C, Xia F, Li W, Navigli R, editors. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… find
  84. Sahoo P, Singh AK, Saha S, Jain V, Mondal S, Chadha A. A systematic survey of prompt engineering in large language models: techniques and applications; 2024. ArXiv:2402.07927.
  85. Settles B.: Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences.
  86. Culotta A, McCallum A. Reducing labeling effort for structured prediction tasks. In: Proceedings of the 20th National Conference on Artificial Intelligence - Volume 2. AAAI’05. AAAI Press; 2005. p. 746–751. find
  87. White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT; 2023. ArXiv:2302.11382.
  88. Desmond M, Brachman M. Exploring prompt engineering practices in the enterprise; 2024. ArXiv:2403.08950. — [arXiv:2403.08950](https://arxiv.org/abs/2403.08950) find
  89. Mondal S, Bappon SD, Roy CK. Enhancing user interaction in ChatGPT: characterizing and consolidating multiple prompts for issue resolution; 2024. ArXiv:2402.04568. — [arXiv:2402.04568](https://arxiv.org/abs/2402.04568) find
  90. White J, Hays S, Fu Q, Spencer-Smith J, Schmidt DC. ChatGPT prompt patterns for improving code quality, refactoring, requirements elicitation, and software design; 2023. ArXiv:2303.07839. — [arXiv:2303.07839](https://arxiv.org/abs/2303.07839) find
  91. Pryzant R, Iter D, Li J, Lee Y, Zhu C, Zeng M. Automatic prompt optimization with “gradient descent” and beam search. In: Bouamor H, Pino J, Bali K, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics;…
  92. Chen Y, Wen Z, Fan G, Chen Z, Wu W, Liu D, et al. MAPO: boosting large language model performance with model-adaptive prompt optimization. In: Bouamor H, Pino J, Bali K, editors. Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguisti… find
  93. Cheng J, Liu X, Zheng K, Ke P, Wang H, Dong Y, et al. Black-box prompt optimization: aligning large language models without model training. In: Ku LW, Martins A, Srikumar V, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bang… find
  94. Wang X, Li C, Wang Z, Bai F, Luo H, Zhang J, et al. PromptAgent: strategic planning with language models enables expert-level prompt optimization; 2023. ArXiv:2310.16427. find
  95. Bach SH, Sanh V, Yong ZX, Webson A, Raffel C, Nayak NV, et al. Promptsource: an integrated development environment and repository for natural language prompts; 2022. ArXiv:2202.01279.
  96. Deng M, Wang J, Hsieh CP, Wang Y, Guo H, Shu T, et al. RLPrompt: optimizing discrete text prompts with reinforcement learning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; 2022. p. 3369–3391.
  97. Awal R, Zhang L, Agrawal A. Investigating prompting techniques for zero- and few-shot visual question answering; 2024. ArXiv:2306.09996. — [arXiv:2306.09996](https://arxiv.org/abs/2306.09996) find
  98. OpenAI.: ChatGPT plugins. Accessed: 2023-10-15. https://openai.com/blog/ chatgpt-plugins. ↗
  99. OpenAI.: GPTs: introducing the latest in conversational AI. Accessed: 2024-05-22. Available from: https://openai.com/index/introducing-gpts/. ↗
  100. Bisson S.: Microsoft build 2023: Microsoft extends its copilots with open standard plugins. Accessed: 2023-05-25. https://www.techrepublic.com/article/ microsoft-extends-copilot-with-open-standard-plugins/. ↗
  101. Ng A.: ChatGPT prompt engineering for developers. Accessed: 2023-07-18. https://www.deeplearning.ai/short-courses/ chatgpt-prompt-engineering-for-developers/. ↗
  102. Roller S, Dinan E, Goyal N, Ju D, Williamson M, Liu Y, et al. Recipes for building an open-domain chatbot. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume; 2021. p. 300–325.
  103. whatplugin.ai.: Prompt enhancer & ChatGPT plugins for AI development tools like prompt enhancer. Accessed: 2023-09-14. https://www.whatplugin.ai/ plugins/prompt-enhancer. ↗
  104. AISEO.ai.: AISEO. Accessed: 2023-8-15. https://aiseo.ai/. ↗
  105. ChatGPT for Search Engines.: Prompt perfect plugin for ChatGPT. Accessed: 2023-10-15. https://chatonai.org/prompt-perfect-chatgpt-plugin. ↗
  106. Prompt Perfect.: Terms of service. Accessed: 2023-09-20. https://promptperfect.xyz/static/terms.html. ↗
  107. Lee K, Firat O, Agarwal A, Fannjiang C, Sussillo D. Hallucinations in neural machine translation; 2018. Https://openreview.net/forum?id=SkxJ-309FQ. find
  108. Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, et al. Survey of hallucination in natural language generation. ACM Computing Surveys. 2023;55(12):1–38.
  109. Lazaridou A, Gribovskaya E, Stokowiec W, Grigorev N. Internet-augmented language models through few-shot prompting for open-domain question answering; 2022. ArXiv:2203.05115.
  110. Jiang Z, Xu FF, Gao L, Sun Z, Liu Q, Dwivedi-Yu J, et al. Active retrieval augmented generation; 2023. ArXiv:2305.06983. find
  111. Ram O, Levine Y, Dalmedigos I, Muhlgay D, Shashua A, Leyton-Brown K, et al. In-context retrieval-augmented language models; 2023. ArXiv:2302.00083. find
  112. Shuster K, Poff S, Chen M, Kiela D, Weston J. Retrieval augmentation reduces hallucination in conversation; 2021. ArXiv:2104.07567.
  113. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems. 2020;33:9459–9474. find
  114. Izacard G, Grave E. Leveraging passage retrieval with generative models for open domain question answering; 2020. ArXiv:2007.01282.
  115. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020. p. 7871–7880.
  116. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 2020;21(1):5485–5551.
  117. Roller S, Dinan E, Goyal N, Ju D, Williamson M, Liu Y, et al. Recipes for building an open-domain chatbot. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume; 2021. p. 300–325.
  118. Dhuliawala S, Komeili M, Xu J, Raileanu R, Li X, Celikyilmaz A, et al. Chain-of-verification reduces hallucination in large language models; 2023. ArXiv:2309.11495.
  119. Li J, Tang T, Zhao WX, Wang J, Nie JY, Wen JR. The web can be your oyster for improving Large Language Models; 2023. ArXiv:2305.10998. — [arXiv:2305.10998](https://arxiv.org/abs/2305.10998) find
  120. Paranjape B, Lundberg S, Singh S, Hajishirzi H, Zettlemoyer L, Ribeiro MT. ART: automatic multi-step reasoning and tool-use for large language models; 2023. ArXiv:2303.09014. find
  121. Greyling C.: 12 prompt engineering techniques. Available from: https://cobusgreyling.medium.com/12-prompt-engineering-techniques-644481c857aa. ↗
  122. Vinija.: Prompt engineering. prompt-engineering/. Available from: https://vinija.ai/nlp/ ↗
  123. Badhan M.: Advanced prompt engineering techniques. Accessed: 2024-05-22. Available from: https://www.mercity.ai/blog-post/ advanced-prompt-engineering-techniques. ↗
  124. Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K, et al. ReAct: synergizing reasoning and acting in language models; 2023. ArXiv:2210.03629.
  125. Li A.: ReAct: a new framework for prompt engineering in large language models. Available from: https://www.perxeive.com/blog/ react-a-new-framework-for-prompt-engineering-in-large-language-models. ↗
  126. Roberts A.: How to ReAct to simple AI agents. Available from: https://arize.com/blog-course/react-agent-llm/. ↗
  127. Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick CL, et al. VQA: visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV); 2015. p. 2425–2433. find
  128. Wu Q, Teney D, Wang P, Shen C, Dick A, van den Hengel A. Visual question answering: a survey of methods and datasets. Computer Vision and Image Understanding. 2017;163:21–40. Language in Vision. find
  129. Wang P, Wu Q, Shen C, Dick A, van den Hengel A. FVQA: fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018;40(10):2413–2427. find
  130. Kafle K, Kanan C. Visual question answering: datasets, algorithms, and future challenges. Computer Vision and Image Understanding. 2017;163:3–20. Language in Vision. find
  131. Yin S, Fu C, Zhao S, Li K, Sun X, Xu T, et al. A survey on multimodal large language models; 2024. ArXiv:2306.13549.
  132. Wu J, Gan W, Chen Z, Wan S, Yu PS. Multimodal large language models: a survey. In: 2023 IEEE International Conference on Big Data; 2023. p. 2247–2256.
  133. Zhang R, Wei Z, Fang R, Gao P, Li K, Dai J, et al. Tip-adapter: training-free adaption of CLIP for few-shot classification; 2022. ArXiv:2207.09519. — [arXiv:2207.09519](https://arxiv.org/abs/2207.09519) find
  134. Ju C, Han T, Zheng K, Zhang Y, Xie W. Prompting visual-language models for efficient video understanding. In: Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV. Berlin, Heidelberg: Springer-Verlag; 2022. p. 105–124. find
  135. Zhou K, Yang J, Loy CC, Liu Z. Learning to prompt for vision-language models. International Journal of Computer Vision. 2022 jul;130(9):2337–2348. find
  136. Jia C, Yang Y, Xia Y, Chen YT, Parekh Z, Pham H, et al. Scaling up visual and vision-language representation learning with noisy text supervision. In: Meila M, Zhang T, editors. Proceedings of the 38th International Conference on Machine Learning. vol. 139 of Proceedings of Machine Learning Research…
  137. Ma C, Liu Y, Deng J, Xie L, Dong W, Xu C. Understanding and mitigating overfitting in prompt tuning for vision-language models; 2023. ArXiv:2211.02219. — [arXiv:2211.02219](https://arxiv.org/abs/2211.02219) find
  138. Agnolucci L, Baldrati A, Todino F, Becattini F, Bertini M, Bimbo AD. ECO: ensembling context optimization for vision-language models; 2023. ArXiv:2307.14063. — [arXiv:2307.14063](https://arxiv.org/abs/2307.14063) find
  139. Chowdhury S, Nag S, Manocha D. APoLLo : unified adapter and prompt learning for vision language models. In: Bouamor H, Pino J, Bali K, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics; 2023. p. 10173… find
  140. Zhou K, Yang J, Loy CC, Liu Z. Conditional prompt learning for vision-language models; 2022. ArXiv:2203.05557. — [arXiv:2203.05557](https://arxiv.org/abs/2203.05557) find
  141. Ma S, Xie CW, Wei Y, Sun S, Fan J, Bao X, et al. Understanding the multi-modal prompts of the pre-trained vision-language model; 2024. ArXiv:2312.11570. — [arXiv:2312.11570](https://arxiv.org/abs/2312.11570) find
  142. Khattak MU, Wasim ST, Naseer M, Khan S, Yang MH, Khan FS. Self-regulating prompts: foundational model adaptation without forgetting; 2023. ArXiv:2307.06948. — [arXiv:2307.06948](https://arxiv.org/abs/2307.06948) find
  143. Khattak MU, Rasheed H, Maaz M, Khan S, Khan FS. MaPLe: multi-modal prompt learning; 2023. ArXiv:2210.03117. — [arXiv:2210.03117](https://arxiv.org/abs/2210.03117) find
  144. Gu J, Han Z, Chen S, Beirami A, He B, Zhang G, et al. A systematic survey of prompt engineering on vision-language foundation models; 2023. ArXiv:2307.12980. — [arXiv:2307.12980](https://arxiv.org/abs/2307.12980) find
  145. Shen L, Tan W, Zheng B, Khashabi D. Flatness-aware prompt selection improves accuracy and sample efficiency; 2023. ArXiv:2305.10713. find
  146. Paul D, Ismayilzada M, Peyrard M, Borges B, Bosselut A, West R, et al. Refiner: reasoning feedback on intermediate representations; 2023. ArXiv:2304.01904.
  147. Ananthakrishnan R, Bhattacharyya P, Sasikumar M, Shah RM. Some issues in automatic evaluation of English-Hindi MT: more blues for BLEU. Icon. 2007;64. find
  148. Callison-Burch C, Osborne M, Koehn P. Re-evaluating the role of BLEU in machine translation research. In: 11th Conference of the European Chapter of the Association for Computational Linguistics; 2006. p. 249–256.
  149. Stent A, Marge M, Singhai M. Evaluating evaluation methods for generation in the presence of variation. In: International Conference on Intelligent Text Processing and Computational Linguistics. Springer; 2005. p. 341–351. find
  150. Adams G, Fabbri A, Ladhak F, Lehman E, Elhadad N. From sparse to dense: GPT-4 summarization with chain of density prompting; 2023. ArXiv:2309.04269. find
  151. Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, et al. Learning to summarize with human feedback. Advances in Neural Information Processing Systems. 2020;33:3008–3021. find
  152. Wang R, Wang H, Mi F, Chen Y, Xue B, Wong KF, et al. Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting; 2023. ArXiv:2305.13733. find
  153. Lee M, Srivastava M, Hardy A, Thickstun J, Durmus E, Paranjape A, et al. Evaluating human-language model interaction; 2022. ArXiv:2212.09746. — [arXiv:2212.09746](https://arxiv.org/abs/2212.09746) find
  154. Papineni K, Roukos S, Ward T, Zhu WJ. BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics; 2002. p. 311–318.
  155. Chin-Yew L. ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out; 2004. p. 74–81.
  156. Banerjee S, Lavie A. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization; 2005. p. 65–72.
  157. Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: evaluating text generation with BERT. In: Ninth International Conference on Learning Representations; 2020. . find
  158. Sai AB, Mohankumar AK, Khapra MM. A survey of evaluation metrics used for NLG systems. ACM Computing Surveys (CSUR). 2022;55(2):1–39. find
  159. Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models; 2023. ArXiv:2206.04615.
  160. Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models; 2022. ArXiv:2206.04615.
  161. Chang Y, Wang X, Wang J, Wu Y, Yang L, Zhu K, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology. 2024;15(3):1–45.
  162. Wang Y, Liu X, Shi S. Deep neural solver for math word problems. In: Proceedings of the 2017 conference on empirical methods in natural language processing; 2017. p. 845–854. find
  163. Qin J, Lin L, Liang X, Zhang R, Lin L. Semantically-aligned universal tree-structured solver for math word problems; 2020. ArXiv:2010.06823. find
  164. Shi S, Wang Y, Lin CY, Liu X, Rui Y. Automatically solving number word problems by semantic parsing and reasoning. In: Proceedings of the 2015 conference on empirical methods in natural language processing; 2015. p. 1132–1142. find
  165. Hosseini MJ, Hajishirzi H, Etzioni O, Kushman N. Learning to Solve Arithmetic Word Problems with Verb Categorization. In: Moschitti A, Pang B, Daelemans W, editors. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computatio…
  166. Roy S, Roth D. Unit dependency graph and its application to arithmetic word problem solving. In: Proceedings of the AAAI conference on artificial intelligence. vol. 31; 2017. .
  167. Koncel-Kedziorski R, Roy S, Amini A, Kushman N, Hajishirzi H. MAWPS: a math word problem repository. In: Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies; 2016. p. 1152–1157. find
  168. Miao SY, Liang CC, Su KY. A diverse corpus for evaluating and developing English math word problem solvers; 2021. ArXiv:2106.15772. find
  169. Goswami M, Sanil V, Choudhry A, Srinivasan A, Udompanyawit C, Dubrawski A. AQuA: a benchmarking tool for label quality assessment; 2024. ArXiv:2306.09467. — [arXiv:2306.09467](https://arxiv.org/abs/2306.09467) find
  170. Amini A, Gabriel S, Lin P, Koncel-Kedziorski R, Choi Y, Hajishirzi H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms; 2019. ArXiv:1905.13319.
  171. Koncel-Kedziorski R, Hajishirzi H, Sabharwal A, Etzioni O, Ang SD. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics. 2015;3:585–597. find
  172. Roy S, Roth D. ArXiv:1608.01413. Solving general arithmetic word problems; 2016.
  173. Hendrycks D, Burns C, Kadavath S, Arora A, Basart S, Tang E, et al. Measuring mathematical problem solving with the math dataset; 2021. ArXiv:2103.03874.
  174. Lightman H, Kosaraju V, Burda Y, Edwards H, Baker B, Lee T, et al. Let’s verify step by step; 2023. ArXiv:2305.20050.
  175. Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, et al. Measuring massive multitask language understanding; 2020. ArXiv:2009.03300.
  176. Thorne J, Vlachos A, Christodoulopoulos C, Mittal A. FEVER: a large-scale dataset for fact extraction and VERification; 2018. ArXiv:1803.05355.
  177. Feng S, Shi W, Bai Y, Balachandran V, He T, Tsvetkov Y. Knowledge card: filling LLMs’ knowledge gaps with plug-in specialized Language Models; 2023. ArXiv:2305.09955. find
  178. Kočiskỳ T, Schwarz J, Blunsom P, Dyer C, Hermann KM, Melis G, et al. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics. 2018;6:317–328.
  179. Pang RY, Parrish A, Joshi N, Nangia N, Phang J, Chen A, et al. QuALITY: question answering with long input texts, yes! In: North American Chapter of the Association for Computational Linguistics; 2021. p. 5336–5358.
  180. Talmor A, Herzig J, Lourie N, Berant J. Commonsenseqa: a question answering challenge targeting commonsense knowledge; 2018. ArXiv:1811.00937.
  181. Talmor A, Herzig J, Lourie N, Berant J. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In: Burstein J, Doran C, Solorio T, editors. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  182. Speer R, Chin J, Havasi C. Conceptnet 5.5: An open multilingual graph of general knowledge. In: Proceedings of the AAAI conference on artificial intelligence. vol. 31; 2017. . find
  183. Yang Z, Qi P, Zhang S, Bengio Y, Cohen WW, Salakhutdinov R, et al. HotpotQA: a dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing; 2018. p. 2369–2380.
  184. Clark P, Cowhey I, Etzioni O, Khot T, Sabharwal A, Schoenick C, et al. Think you have solved question answering? try arc, the ai2 reasoning challenge; 2018. ArXiv:1803.05457. find
  185. Huang L, Cao S, Parulian N, Ji H, Wang L. Efficient attentions for long document summarization; 2021. ArXiv:2104.02112.
  186. Voorhees EM, Tice DM. Building a question answering test collection. In: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval; 2000. p. 200–207. find
  187. Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng AY, et al. Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing; 2013. p. 1631–1642. find
  188. Chen M, Chu Z, Wiseman S, Gimpel K. Summscreen: a dataset for abstractive screenplay summarization; 2021. ArXiv:2104.07091.
  189. Zhang X, Zhao J, LeCun Y. Character-level convolutional networks for text classification. Advances in neural information processing systems. 2015;28.
  190. Zhang W, Deng Y, Liu B, Pan SJ, Bing L. Sentiment analysis in the era of large language models: a reality check; 2023. ArXiv:2305.15005.
  191. Hu M, Liu B. Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining; 2004. p. 168–177. find
  192. Pang B, Lee L. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales; 2005. Cs/0506075.
  193. Tang L, Peng Y, Wang Y, Ding Y, Durrett G, Rousseau JF. Less likely brainstorming: Using language models to generate alternative hypotheses. In: Proceedings of the conference. Association for Computational Linguistics. Meeting. vol. 2023. NIH Public Access; 2023. p. 12532. find
  194. Raunak V, Post M, Menezes A. SALTED: a framework for SAlient long-tail translation error detection; 2022. ArXiv:2205.09988. find
  195. Yu L, Poirson P, Yang S, Berg AC, Berg TL. Modeling context in referring expressions. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer; 2016. p. 69–85.
  196. Mao J, Huang J, Toshev A, Camburu O, Yuille AL, Murphy K. Generation and comprehension of unambiguous object descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 11–20.
  197. Jain N, Saifullah K, Wen Y, Kirchenbauer J, Shu M, Saha A, et al. Bring your own data! Self-supervised evaluation for large language models; 2023. ArXiv:2306.13651. — [arXiv:2306.13651](https://arxiv.org/abs/2306.13651) find
  198. Wang Y, Yu Z, Zeng Z, Yang L, Wang C, Chen H, et al. PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization; 2023. ArXiv:2306.05087.
  199. Lin YT, Chen YN. LLM-eval: unified multi-dimensional automatic evaluation for open-domain conversations with large language models; 2023. ArXiv:2305.13711.
  200. Dehghani M, Tay Y, Gritsenko AA, Zhao Z, Houlsby N, Diaz F, et al. The benchmark lottery; 2021. ArXiv:2107.07002. find
  201. Kiela D, Bartolo M, Nie Y, Kaushik D, Geiger A, Wu Z, et al. Dynabench: rethinking benchmarking in NLP; 2021. ArXiv:2104.14337. find
  202. Xu W, Banburski-Fahey A, Jojic N. Reprompting: automated chain-of-thought prompt inference through gibbs sampling; 2023. ArXiv:2305.09993. — [arXiv:2305.09993](https://arxiv.org/abs/2305.09993) find
  203. Wang L, Xu W, Lan Y, Hu Z, Lan Y, Lee RKW, et al. Plan-and-solve prompting: improving zero-shot chain-of-thought reasoning by large language models; 2023. ArXiv:2305.04091. find
  204. Mishra S, Khashabi D, Baral C, Hajishirzi H. Cross-task generalization via natural language crowdsourcing instructions; 2021. ArXiv:2104.08773. find
  205. Pryzant R, Iter D, Li J, Lee YT, Zhu C, Zeng M. Automatic prompt optimization with ”gradient descent” and beam search; 2023. ArXiv:2305.03495.
  206. Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, et al. Large Language Models are human-level prompt engineers. In: Eleventh International Conference on Learning Representations; 2022. .
  207. Chen H, Pasunuru R, Weston J, Celikyilmaz A. Walking down the memory maze: beyond context limit through interactive reading; 2023. ArXiv:2310.05029. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) find
  208. Chevalier A, Wettig A, Ajith A, Chen D. Adapting language models to compress contexts; 2023. ArXiv:2305.14788.
  209. Bulatov A, Kuratov Y, Kapushev Y, Burtsev MS. Scaling transformer to 1m tokens and beyond with rmt; 2023. ArXiv:2304.11062. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) find
  210. Xu J, Szlam A, Weston J. Beyond goldfish memory: Long-term open-domain conversation; 2021. ArXiv:2107.07567. find
  211. Wu Y, Rabe MN, Hutchins D, Szegedy C. Memorizing transformers; 2022. ArXiv:2203.08913. find
  212. Izacard G, Grave E. Leveraging passage retrieval with generative models for open domain question answering; 2020. ArXiv:2007.01282.
  213. Guo Q, Wang R, Guo J, Li B, Song K, Tan X, et al. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers; 2023. ArXiv:2309.08532. find
  214. Sridhar A, Lo R, Xu FF, Zhu H, Zhou S. Hierarchical prompting assists large language model on web navigation; 2023. ArXiv:2305.14257. find
  215. Yao S, Chen H, Yang J, Narasimhan K. Webshop: towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems. 2022;35:20744–20757.
  216. Zhang C, Xiao J, Chen L, Shao J, Chen L. TreePrompt: Learning to Compose Tree Prompts for Explainable Visual Grounding; 2023. ArXiv:2305.11497. — [arXiv:2305.11497](https://arxiv.org/abs/2305.11497) find
  217. Colangeli M, Di Francesco A, Rondoni L. Finite reservoirs and irreversibility corrections to Hamiltonian systems statistics; 2023. ArXiv:2305.14922. — [arXiv:2305.14922](https://arxiv.org/abs/2305.14922) find
  218. Jiang H, Wu Q, Lin CY, Yang Y, Qiu L. Llmlingua: Compressing prompts for accelerated inference of large language models; 2023. ArXiv:2310.05736.
  219. Ning X, Lin Z, Zhou Z, Yang H, Wang Y. Skeleton-of-thought: Large language models can do parallel decoding; 2023. ArXiv:2307.15337. find
  220. Hu H, Lu H, Zhang H, Song YZ, Lam W, Zhang Y. Chain-of-symbol prompting elicits planning in Large Langauge Models; 2023. ArXiv:2305.10276. — [arXiv:2305.10276](https://arxiv.org/abs/2305.10276) find
  221. Feng S, Shi W, Bai Y, Balachandran V, He T, Tsvetkov Y. Knowledge Card: Filling LLMs’ Knowledge Gaps with Plug-in Specialized Language Models; 2023. ArXiv:2305.09955. find
  222. Krishna S, Ma J, Slack D, Ghandeharioun A, Singh S, Lakkaraju H. Post hoc explanations of language models can improve language models. Advances in Neural Information Processing Systems. 2024;36. find
  223. Sun S, Liu Y, Wang S, Zhu C, Iyyer M. Pearl: Prompting large language models to plan and execute actions over long documents; 2023. ArXiv:2305.14564. find
  224. Chen W, Ma X, Wang X, Cohen WW. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks; 2022. ArXiv:2211.12588. find
  225. Press O, Zhang M, Min S, Schmidt L, Smith NA, Lewis M. Measuring and narrowing the compositionality gap in language models; 2022. ArXiv:2210.03350. find
  226. Schick T, Dwivedi-Yu J, Dessı̀ R, Raileanu R, Lomeli M, Hambro E, et al. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems. 2024;36.
  227. Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, et al. Large Language Models Are Human-Level Prompt Engineers; 2023. ArXiv:2211.01910.
  228. Ajith A, Pan C, Xia M, Deshpande A, Narasimhan K. InstructEval: systematic evaluation of instruction selection methods; 2023. ArXiv:2307.00259. find
  229. Tang J, Zhou X, Wan X, Daley M, Bai Z. ML4STEM professional development program: enriching K-12 STEM teaching with machine learning. International Journal of Artificial Intelligence in Education. 2023;33(1):185–224. find
  230. Xie Q, Dai Z, Hovy E, Luong MT, Le QV. Unsupervised data augmentation for consistency training. In: Advances in neural information processing systems. vol. 33; 2020. p. 6256–6268.
  231. Ariely M, Nazaretsky T, Alexandron G. Machine learning and Hebrew NLP for automated assessment of open-ended questions in biology. International Journal of Artificial Intelligence in Education. 2023;33(1):1–34. find
  232. Nilsson F, Tuvstedt J. GPT-4 as an automatic grader: the accuracy of grades set by GPT-4 on introductory programming assignments [Bachelor Thesis]. KTH Royal Institute of Technology; 2023. find
  233. Schneider J, Richner R, Riser M. Towards trustworthy autograding of short, multi-lingual, multi-type answers. International Journal of Artificial Intelligence in Education. 2023;33(1):88–118. find
  234. Yang K, Tian Y, Peng N, Klein D. Re3: generating longer stories with recursive reprompting and revision. In: Goldberg Y, Kozareva Z, Zhang Y, editors. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computationa… find
  235. Yang K, Klein D, Peng N, Tian Y. DOC: improving long story coherence with detailed outline control. In: Rogers A, Boyd-Graber J, Okazaki N, editors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Compu…
  236. Yang K, Klein D. FUDGE: controlled text generation with future discriminators. In: Toutanova K, Rumshisky A, Zettlemoyer L, Hakkani-Tur D, Beltagy I, Bethard S, et al., editors. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human L…
  237. Elgohary A, Hosseini S, Awadallah AH. Speak to your parser: interactive text-to-SQL with natural language feedback. In: Annual Meeting of the Association for Computational Linguistics; 2020. p. 2065–2077. find
  238. Nijkamp E, Pang B, Hayashi H, Tu L, Wang H, Zhou Y, et al. Codegen: an open large language model for code with multi-turn program synthesis; 2022. ArXiv:2203.13474. find
  239. Shrivastava D, Larochelle H, Tarlow D. Repository-level prompt generation for large language models of code. In: International Conference on Machine Learning; 2023. p. 31693–31715. find
  240. Shwartz V, West P, Bras RL, Bhagavatula C, Choi Y. Unsupervised commonsense question answering with self-talk; 2020. ArXiv:2004.05483. find
  241. Uesato J, Kushman N, Kumar R, Song F, Siegel N, Wang L, et al. Solving math word problems with process-and outcome-based feedback; 2022. ArXiv:2211.14275.
  242. Roy S, Roth D. Solving general arithmetic word problems. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; 2015. p. 1743–1752.
  243. Li Y, Lin Z, Zhang S, Fu Q, Chen B, Lou JG, et al. Making language models better reasoners with step-aware verifier. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics; 2023.…
  244. Ding B, Qin C, Liu L, Bing L, Joty S, Li B. Is GPT-3 a good data annotator?; 2022. ArXiv:2212.10450.
  245. Yoo KM, Park D, Kang J, Lee SW, Park W. GPT3Mix: leveraging large-scale language models for text augmentation. In: Findings of the Association for Computational Linguistics: EMNLP 2021; 2021. p. 2225–2239.
  246. Liu Y, Deng G, Xu Z, Li Y, Zheng Y, Zhang Y, et al. Jailbreaking ChatGPT via prompt engineering: an empirical study; 2024. ArXiv:2305.13860.
  247. Yao Y, Duan J, Xu K, Cai Y, Sun Z, Zhang Y. A survey on Large Language Model (LLM) security and privacy: the good, the bad, and the ugly. High-Confidence Computing. 2024 jun;4(2):100211.
  248. Rawat P.: AI at risk: OWASP top 10 critical vulnerabilities for large language models (LLMs). Available from: https://www.infosectrain.com/blog/ ai-at-risk-owasp-top-10-critical-vulnerabilities-for-large-language-models-llms/. ↗
  249. Perez F, Ribeiro I. Ignore previous prompt: attack techniques for language models; 2022. ArXiv:2211.09527.
  250. Ren K, Zheng T, Qin Z, Liu X. Adversarial attacks and defenses in deep learning. Engineering. 2020;6(3):346–360. find
  251. Yin Z, Ye M, Zhang T, Du T, Zhu J, Liu H, et al. VLATTACK: multimodal adversarial attacks on vision-language tasks via pre-trained models; 2024. ArXiv:2310.04655. — [arXiv:2310.04655](https://arxiv.org/abs/2310.04655) find
  252. Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples; 2015. ArXiv:1412.6572.
  253. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition; 2009. p. 248–255. find
  254. Wang J, Liu Z, Park KH, Jiang Z, Zheng Z, Wu Z, et al. Adversarial demonstration attacks on large language models; 2023. ArXiv:2305.14950. — [arXiv:2305.14950](https://arxiv.org/abs/2305.14950) find
  255. Kolter Z.: Discrete optimization for adversarial attacks on large language models. Available from: https://orc.mit.edu/events/ discrete-optimization-adversarial-attacks-large-language-models. ↗
  256. Shayegani E, Mamun MAA, Fu Y, Zaree P, Dong Y, Abu-Ghazaleh N. Survey of vulnerabilities in large language models revealed by adversarial attacks; 2023. ArXiv:2310.10844.
  257. Selvakkumar A, Pal S, Jadidi Z. Addressing adversarial machine learning attacks in smart healthcare perspectives; 2021. ArXiv:2112.08862. — [arXiv:2112.08862](https://arxiv.org/abs/2112.08862) find
  258. Bagdasaryan E, Veit A, Hua Y, Estrin D, Shmatikov V. How to backdoor federated learning. In: Chiappa S, Calandra R, editors. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics. vol. 108 of Proceedings of Machine Learning Research. PMLR; 2020. p. 2938–… find
  259. Steinhardt J, Koh PW, Liang P. Certified defenses for data poisoning attacks; 2017. ArXiv:1706.03691. — [arXiv:1706.03691](https://arxiv.org/abs/1706.03691) find
  260. Yang H, Xiang K, Ge M, Li H, Lu R, Yu S. A comprehensive overview of backdoor attacks in large language models within communication networks. IEEE Network. 2024;p. 1–1. find
  261. Chen X, Liu C, Li B, Lu K, Song D. Targeted backdoor attacks on deep learning systems using data poisoning; 2017. ArXiv:1712.05526. — [arXiv:1712.05526](https://arxiv.org/abs/1712.05526) find
  262. Shamshiri S, Sohn I. Defense method challenges against backdoor attacks in neural networks. In: 2024 International Conference on Artificial Intelligence in Information and Communication; 2024. p. 396–400. find
  263. Holland R, Pal S, Pan L, Zhang LY. Backdoor attacks and generative model fairness: current trends and future research directions. In: 2024 16th International Conference on COMmunication Systems & NETworkS; 2024. p. 31–36. find
  264. Khan A, Sharma I. AI-powered detection and mitigation of backdoor attacks on databases server. In: 2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things; 2024. p. 374–379. find
  265. Ooi YX. Evaluation of backdoor attacks and defenses to deep neural networks [Master’s thesis]. Nanyang Technological University; 2024. find
  266. Li Y, Li T, Chen K, Zhang J, Liu S, Wang W, et al. BadEdit: backdooring large language models by model editing; 2024. ArXiv:2403.13355. — [arXiv:2403.13355](https://arxiv.org/abs/2403.13355) find
  267. Gu T, Dolan-Gavitt B, Garg S. BadNets: identifying vulnerabilities in the machine learning model supply chain; 2019. ArXiv:1708.06733. find
  268. Liu Y, Ma X, Bailey J, Lu F. Reflection backdoor: a natural backdoor attack on deep neural networks. In: Computer Vision – ECCV 2020. Berlin, Heidelberg: Springer-Verlag; 2020. p. 182–199. find
  269. Zhao S, Wen J, Luu A, Zhao J, Fu J. Prompt as triggers for backdoor attack: examining the vulnerability in language models. In: Bouamor H, Pino J, Bali K, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguis… find
  270. Chow KH, Wei W, Yu L. Imperio: language-guided backdoor attacks for arbitrary model control; 2024. ArXiv:2401.01085. — [arXiv:2401.01085](https://arxiv.org/abs/2401.01085) find
  271. Abdali S, Anarfi R, Barberan C, He J. Securing large language models: threats, vulnerabilities and responsible practices; 2024. ArXiv:2403.12503. — [arXiv:2403.12503](https://arxiv.org/abs/2403.12503) find
  272. Kosch T, Feger S. Risk or chance? Large language models and reproducibility in human-computer interaction research; 2024. ArXiv:2404.15782. — [arXiv:2404.15782](https://arxiv.org/abs/2404.15782) find
  273. Liu FW, Hu C. Exploring vulnerabilities and protections in large language models: a survey; 2024. ArXiv:2406.00240. — [arXiv:2406.00240](https://arxiv.org/abs/2406.00240)
  274. Zhan Q, Liang Z, Ying Z, Kang D. InjecAgent: benchmarking indirect prompt injections in tool-integrated large language model agents; 2024. ArXiv:2403.02691.
  275. Chen K, Wang Z, Mi B, Liu W, Wang S, Ren X, et al. Machine unlearning in large language models; 2024. ArXiv:2404.16481. — [arXiv:2404.16481](https://arxiv.org/abs/2404.16481) find
  276. Wang H, Li H, Huang M, Sha L. From noise to clarity: unraveling the adversarial suffix of large language model attacks via translation of text embeddings; 2024. ArXiv:2402.16006. — [arXiv:2402.16006](https://arxiv.org/abs/2402.16006) find
  277. Gao A. Prompt engineering for large language models; 2023. SSRN:4504303. find
  278. Learn prompting.: Introduction to prompt hacking. Available from: https://learnprompting.org/docs/prompt hacking/intro. ↗
  279. Schulhoff S, Pinto J, Khan A, Bouchard LF, Si C, Anati S, et al. Ignore this title and HackAPrompt: exposing systemic vulnerabilities of LLMs through a global prompt hacking competition. In: Bouamor H, Pino J, Bali K, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Langua… find
  280. Allouin A.: Understanding LLM prompt hacking and attacks. Available from: https://medium.com/@alexandre.allouin/ understanding-llm-prompt-hacking-and-attacks-8781c313a25b. ↗
  281. Karande C.: OWASP LLM prompt hacking. Available from: https://owasp.org/ www-project-llm-prompt-hacking/. ↗
  282. Teneo AI.: Navigating the challenges of prompt hacking in 2024. Available from: https://www.teneo.ai/blog/ navigating-the-challenges-of-prompt-hacking-in-2024. ↗
  283. Krishna K, Tomar GS, Parikh AP, Papernot N, Iyyer M. Thieves on sesame street! Model extraction of BERT-based APIs. In: Ninth International Conference on Learning Representations; 2020. . find
  284. Papernot N, McDaniel P, Sinha A, Wellman MP. SoK: security and privacy in machine learning. In: 2018 IEEE European Symposium on Security and Privacy (EuroS&P); 2018. p. 399–414. find
  285. Tramèr F, Zhang F, Juels A, Reiter MK, Ristenpart T. Stealing machine learning models via prediction APIs. In: Proceedings of the 25th USENIX Conference on Security Symposium. SEC’16. USA: USENIX Association; 2016. p. 601–618. find
  286. Shen X, Qu Y, Backes M, Zhang Y. Prompt stealing attacks against text-to-image generation models; 2023. ArXiv:2302.09923. — [arXiv:2302.09923](https://arxiv.org/abs/2302.09923) find
  287. Zhang S. Defending against model extraction attacks via watermark-based method with knowledge distillation [Master’s thesis]. Nanyang Technological University; 2024. find
  288. Naseh A, Krishna K, Iyyer M, Houmansadr A. Stealing the decoding algorithms of language models. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. CCS ’23. New York, NY, USA: Association for Computing Machinery; 2023. p. 1835–1849. find
  289. Carlini N, Paleka D, Dvijotham KD, Steinke T, Hayase J, Cooper AF, et al. Stealing part of a production language model; 2024. ArXiv:2403.06634. — [arXiv:2403.06634](https://arxiv.org/abs/2403.06634) find
  290. Sha Z, Zhang Y. Prompt stealing attacks against large language models; 2024. ArXiv:2402.12959. — [arXiv:2402.12959](https://arxiv.org/abs/2402.12959) find
  291. Yang Y, Li C, Jiang Y, Chen X, Wang H, Zhang X, et al. PRSA: PRompt Stealing Attacks against large language models; 2024. ArXiv:2402.19200. — [arXiv:2402.19200](https://arxiv.org/abs/2402.19200) find
  292. Hu H, Salcic Z, Sun L, Dobbie G, Yu PS, Zhang X. Membership inference attacks on machine learning: a survey; 2022. ArXiv:2103.07853. — [arXiv:2103.07853](https://arxiv.org/abs/2103.07853) find
  293. Kurakin A, Goodfellow I, Bengio S. Adversarial machine learning at scale; 2017. ArXiv:1611.01236. — [arXiv:1611.01236](https://arxiv.org/abs/1611.01236) find
  294. Bai T, Luo J, Zhao J, Wen B, Wang Q. Recent advances in adversarial training for adversarial robustness. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization; 2021. p. 4312–4321. Survey Tra… find
  295. Eric M.: A complete introduction to prompt engineering for large language models. Available from: https://www.mihaileric.com/posts/ a-complete-introduction-to-prompt-engineering/. ↗
  296. Linardatos P, Papastefanopoulos V, Kotsiantis S. Explainable AI: a review of machine learning interpretability methods. Entropy. 2020;23(1):18. find
  297. Recht B, Re C, Wright S, Niu F. Hogwild!: a lock-free approach to parallelizing stochastic gradient descent. In: Advances in neural information processing systems. vol. 24; 2011. . find
  298. Melnychuk V, Frauen D, Feuerriegel S. Causal transformer for estimating counterfactual outcomes. In: International Conference on Machine Learning; 2022. p. 15293–15329. find
  299. Novakovsky G, Dexter N, Libbrecht MW, Wasserman WW, Mostafavi S. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nature Reviews Genetics. 2023;24(2):125–137. find
  300. Bertucci L, Brière M, Fliche O, Mikael J, Szpruch L. Deep learning in finance: from implementation to regulation; 2022. SSRN:4080171. find
  301. Maple C, Szpruch L, Epiphaniou G, Staykova K, Singh S, Penwarden W, et al. The AI revolution: opportunities and challenges for the finance sector; 2023. ArXiv:2308.16538. — [arXiv:2308.16538](https://arxiv.org/abs/2308.16538) find
  302. Amann J, Blasimme A, Vayena E, Frey D, Madai VI. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making. 2020;20(1):1–9. find
  303. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nature Medicine. 2022;28(1):31–38. find
  304. Öztürk D. What does artificial intelligence mean for organizations? A systematic review of organization studies research and a way forward. The Impact of Artificial Intelligence on Governance, Economics and Finance, Volume I. 2021;p. 265–289. find
  305. Seeamber R, Badea C. If our aim is to build morality into an artificial agent, how might we begin to go about doing so? IEEE Intelligent Systems. 2023;. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-unleashing-potential-prompt-engineering-large/