Elamur Research⌕
☰
home / papers / 2024 / The Effect of Sampling Temperature on Problem Solving in Large Language Models

The Effect of Sampling Temperature on Problem Solving in Large Language Models

Paper · 2024 · Matthew Renze

Summary

Practitioners often pick an LLM's sampling temperature by intuition: low for precise work, high for "creative" work, with little evidence behind either. Renze and Guven test this systematically for problem solving. They built multiple-choice (MCQA) exams by randomly drawing questions from ten problem sets covering science, math, common sense, logic, law, medicine and English. They then ran nine LLMs, each with five system-prompt styles, and swept temperature from 0.0 upward. Every question was answered ten times so that average accuracy could be measured.

They found no statistically significant change in accuracy across the 0.0–1.0 range. This held across models, prompting styles and domains. Beyond 1.0, performance eventually falls off. The authors therefore recommend temperature 0.0 for problem-solving tasks, since it gives the most reproducible outputs at no cost in accuracy. The result can save engineers tuning effort. It also suggests that moderate temperatures neither trigger answer-changing hallucinations nor help the model search for correct MCQA solutions.

Key points

  • Models: nine LLMs, grouped by family: GPT-3.5 Turbo and GPT-4; Gemini 1.0 Pro and Gemini 1.5 Pro (Preview); Llama 2 7B Chat and Llama 2 70B Chat; plus Claude 3 Opus, Mistral Large and Command R+. Only models whose API exposes temperature were eligible.
  • Prompting styles: five, all one-shot and single-turn:
    • Baseline
    • Domain Expertise
    • Self-recitation
    • Chain-of-Thought
    • Composite (adds self-criticism)
  • Data: the ten problem sets are ARC Challenge, AQUA-RAT, Hellaswag, LogiQA, LSAT-AR/LR/RC, MedMCQA, SAT-English and SAT-Math. They were drawn from the ARC, AGI Eval, Hellaswag and MedMCQA benchmarks and converted to a common JSON-L format.
  • Two exam sizes:
    • A 1,000-question exam (100 per set) was used with GPT-3.5 for the per-domain analysis.
    • A 100-question exam (10 per set) was used across all models and prompts.
    • Only GPT-3.5 with CoT on the small exam was pushed to temperature 1.6.
  • Controls and costs: other sampling parameters (top-p, top-k, repetition penalty) were held fixed. At high temperatures, responses grow much longer, which drives up cost and runtime.

Interesting ideas

  • I1. Normalize benchmarks into one schema. Converting heterogeneous benchmarks into a single JSON-L format lets the test harness swap exams without code changes. This is useful for any multi-dataset evaluation, such as mixing Ukrainian and English test sets.
  • I2. Repeat each query several times. Asking each question ten times per configuration separates real effects from sampling noise. Any study of stochastic decoding should measure variance, not just a single-run accuracy.
  • I3. Budget the experiment grid deliberately. The authors define the full factorial design (model × prompt × exam × temperature × problem × attempt) but run only a cost-aware subset. A small exam covers breadth, and a larger exam is reserved for one model to get depth. This is a practical way to plan expensive API experiments.
  • I4. Default to temperature 0.0 for tasks with one correct answer. This gives reproducibility at no accuracy cost, which matters for annotation pipelines and evaluation runs that need to be repeated.
  • I5. Hold other parameters fixed. Isolating a single hyperparameter by freezing all the others is simple, but it is what makes the conclusions attributable to temperature alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksReasoning · 510 works

Cite

Cite as Matthew Renze. 2024. The Effect of Sampling Temperature on Problem Solving in Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.432

@inproceedings{renze2024effect,
  title = {{The Effect of Sampling Temperature on Problem Solving in Large Language Models}},
  author = {Matthew Renze},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.432},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.432}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science, 9:147–169. find
  2. Anthropic. 2024a. The claude 3 model family: Opus, sonnet, haiku. find
  3. Anthropic. 2024b. Introducing the next generation of claude anthropic. find
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. ArXiv. find
  5. Cohere. 2024a. Command r+. find
  6. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213.
  7. William H. Kruskal and W. Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583– 621. find
  8. Minhyeok Lee. 2023. A mathematical investigation of hallucination and creativity in GPT models. Mathematics, 11:2320. find
  9. V.I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10:707–710. find
  10. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In International Joint Conference on Artificial Intelligence.
  11. Llama-2-Team. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv. find
  12. Meta. 2023. Meta and microsoft introduce the next generation of llama \| meta. find
  13. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey. arXiv.
  14. Cohere. 2024b. Model card for c4ai command r+. find
  15. Microsoft. 2023. Completions - learn how to generate or manipulate text. find
  16. Olive Jean Dunn. 1964. Multiple comparisons using rank sums. Technometrics, 6(3):241–252. find
  17. Mistral-AI-Team. 2024. Au large \| mistral ai \| frontier ai in your hands. find
  18. Gemini-Team. 2023. Gemini: A family of highly capable multimodal models. arXiv.
  19. OpenAI. 2022a. Introducing ChatGPT. find
  20. Gemini-Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
  21. OpenAI. 2023a. Api temperature change from 0 to 1 to 0 to 2 in ’playground’. find
  22. Zellig S. Harris. 1954. Distributional structure. WORD, 10:146–162. find
  23. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv.
  24. Siqing Huo, Negar Arabzadeh, and Charles L. A. Clarke. 2023. Retrieving supporting evidence for generative question answering. arXiv. find
  25. Paul Jaccard. 1912. The distribution of flora in the alpine zone. New Phytologist, 11:37–50. find
  26. Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21. find
  27. OpenAI. 2022b. Models - openai api. find
  28. OpenAI. 2023c. GPT-4 technical report. arXiv.
  29. OpenAI. 2023d. OpenAI - API reference. find
  30. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, pages 248–260. PMLR.
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, page 311. Association for Computational Linguistics. find
  32. Sundar Pichai and Demis Hassabis. 2023. Introducing gemini: Google’s most capable ai model yet. find
  33. Sundar Pichai and Demis Hassabis. 2024. Introducing gemini 1.5, google’s next-generation ai model. find
  34. Vinay Pursnani, Yusuf Sermet, and Ibrahim Demir. 2023. Performance of ChatGPT on the US fundamentals of engineering exam: Comprehensive assessment of proficiency and potential implications for professional environmental engineering practice. arXiv. find
  35. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  37. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. AGIEval: A human-centric benchmark for evaluating foundation models. ArXiv.
  38. Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Jia Li, Zhi Jin, and Hong Mei. 2023. Improving code generation by dynamic temperature sampling. arXiv. find
  39. Jessica Shieh. 2024. Best practices for prompt engineering with OpenAI API. find
  40. Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations. find
  41. Chi Wang, Susan Xueqing Liu, and Ahmed H Awadallah. 2023a. Cost-effective hyperparameter optimization for large language model generation inference. find
  42. Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, and Da-Chang Juan. 2020. Contextual temperature for language modeling. arXiv. find
  43. Rui Wang, Hongru Wang, Fei Mi, Yi Chen, Ruifeng Xu, and Kam-Fai Wong. 2023b. Self-critique prompting with large language models for inductive instructions. arXiv. find
  44. Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2021. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech and Language Processing, 30:2201–2216. find
  45. Ian Ward. 2014. JSON lines. find
  46. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv.
  47. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv.
  48. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/renze-2024-effect-sampling-temperature-problem-solving/