Elamur Research⌕
☰
home / papers / 2023 / Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Paper · 2023 · Yao Fu, Litu Ou, Mingyu Chen et al.

Summary

Chat-style comparisons make many LLMs look alike, so the authors argue that multi-step reasoning is a better way to separate strong models from weak ones. It also matters for using LLMs as engines behind real applications. They introduce Chain-of-Thought Hub (CoT Hub), an open-source and continuously updated evaluation suite. It collects six demanding benchmarks: GSM8k, MATH, MMLU, BigBench Hard, HumanEval and C-Eval. Models are tested with few-shot chain-of-thought prompting, and only final-answer accuracy is scored. Results come from the original papers where available, and the authors ran their own scripts for the rest.

Across 19 models from the GPT, Claude, PaLM, LLaMA and FlanT5 families, three patterns emerge:

  • Reasoning scores rise roughly log-linearly with model size.
  • Closed models (GPT-4, then claude-v1.3 and PaLM-2) clearly lead open ones.
  • Most top-ranked systems have gone through RLHF.

Among open models, LLaMA 65B comes close to code-davinci-002, the base of GPT-3.5. This suggests open-source work should focus on stronger base models and RLHF.

Key points

  • The suite spans 6 benchmarks and over 100 subtasks, including Chinese via C-Eval. Code and results are at github.com/FranxYao/chain-of-thought-hub.
  • Few-shot CoT prompting is chosen over zero-shot so that base and instruction-tuned checkpoints are compared fairly. This sets it apart from HeLM, which mostly uses answer-only prompts.
  • Models are ranked by GSM8k. GPT-4 leads with 92.0, followed by claude-v1.3 (81.8) and PaLM-2 (80.7). LLaMA 65B reaches 50.9.
  • On MMLU, LLaMA 65B scores 63.4, close to code-davinci-002 at 64.5. MATH remains very hard, with GPT-4 at only 42.5%.
  • Five of the top six models are RLHF-tuned, PaLM-2 being the exception. Planned additions include commonsense, theorem-proving and tool-use datasets, plus more models such as Vicuna and Cohere.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksReasoning · 510 works

Cite

Cite as Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv:2305.17306.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.17306 · arXiv 2305.17306

@misc{fu2023chain,
  title = {{Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance}},
  author = {Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng and Tushar Khot},
  year = {2023},
  doi = {10.48550/arxiv.2305.17306},
  eprint = {2305.17306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.17306}
}

References

25 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. models to their sources. Yao Fu’s Notion, Dec 2022. URL https://yaofu.notion.site/ How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint ar… ↗
  2. Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022.
  3. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find
  4. Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley.edu/ blog/2023/04/03/koala/. ↗
  5. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. find
  6. Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022.
  7. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.
  8. Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023.
  9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  10. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  11. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/ 2023-03-30-vicuna/. ↗
  12. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
  13. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  14. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  15. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  16. Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.
  17. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  18. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022.
  19. Databricks. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Blog post, April 2023. URL https://www.databricks.com/blog/2023/04/12/ Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., dolly-first-open-commercially-viable-instruction-tuned-llm. Yasunaga, M., Zhang,… ↗
  20. OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023a. arXiv preprint OpenAI. Gpt-4, 2023b. URL https://openai.com/ research/gpt-4. ↗
  21. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  22. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca, 2023. ↗
  23. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  24. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
  25. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/