Elamur Research⌕
☰
home / papers / 2023 / ThoughtSource: A central hub for large language model reasoning data

ThoughtSource: A central hub for large language model reasoning data

Paper · 2023 · Simon Ott, Konstantin Hebenstreit, Valentin Lievin et al.

Summary

Large language models often stumble on multi-step reasoning, and their decision process is hard to inspect. Chain-of-thought (CoT) prompting, where a model spells out intermediate steps in natural language, partly addresses this, but CoT data has been scattered across incompatible sources. ThoughtSource is an open meta-dataset and Python toolkit that brings this material together. The authors chose QA and inference benchmarks that already contained rationales or explanations. They wrote Hugging Face datasets loaders for each one and offer two views of every dataset: the original source schema and a unified ThoughtSource schema derived from BigBIO's QA schema. Dataset-specific converters turn expert or crowdsourced explanations into "reference CoTs" by linearizing fact lists, entailment trees and structured explanations, fixing punctuation and capitalization, and undoing tokenization. The collection also includes model-generated CoTs, some imported from earlier studies and some newly produced. Additional tools, built on LangChain, generate CoTs through LLM APIs, evaluate them, and support annotation. The resource gives researchers a common basis for analyzing, benchmarking and training reasoning in LLMs.

Key points

  • The first release covers 15 datasets: seven scientific/medical (WorldTree V2, EntailmentBank, OpenBookQA, MedQA (USMLE), MedMCQA, PubmedQA, MMLU medical subsets), three general-domain (CommonsenseQA, StrategyQA, QED) and five math word problem sets (AQUA-RAT, ASDiv, GSM8K, SVAMP, MAWPS).
  • Reference CoTs exist for every dataset except PubmedQA and MedQA. Extra CoTs come from ECQA, few-shot prompting (Wei et al., Liévin et al.) and zero-shot prompting (Kojima et al.). A ThoughtSource-33 subset adds new zero-shot CoTs from several LLMs.
  • A curated library of prompt fragments (instructions, CoT triggers such as "Let's think step by step", answer-extraction templates) makes it possible to test systematically how prompt wording affects results.
  • Two web tools are included: a Dataset Viewer for browsing and an Annotator that highlights similar text across CoTs and records judgments on individual chains.
  • Code and generated data are MIT-licensed and hosted at github.com/OpenBioLink/ThoughtSource, with a Zenodo archive. Adapted data keeps the licenses of its original sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KTH Royal Institute of TechnologyStockholm · Sweden18 works · 788 citations🎓Medical University of ViennaVienna · Austria3 works · 161 citations🎓Technical University of DenmarkKongens Lyngby · Denmark4 works · 698 citations🏥Copenhagen University HospitalCopenhagen · Denmark4 works · 1,008 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Simon Ott, Konstantin Hebenstreit, Valentin Lievin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv:2301.11596 · Scientific Data 10, 528 (2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41597-023-02433-3 · arXiv 2301.11596

@misc{ott2023thoughtsource,
  title = {{ThoughtSource: A central hub for large language model reasoning data}},
  author = {Simon Ott, Konstantin Hebenstreit, Valentin Lievin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther and Matthias Samwald},
  year = {2023},
  doi = {10.1038/s41597-023-02433-3},
  eprint = {2301.11596},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41597-023-02433-3}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Sanh, V. et al. Multitask Prompted Training Enables Zero-Shot Task Generalization. Preprint at https://doi.org/10.48550/arXiv.2110.08207 (2021). \#\# 2. Brown, T. B. et al. Language Models are Few-Shot Learners. Preprint at https://doi.org/10.48550/arXiv.2005.14165 (2020). \#\# 3. Ouyang, L. et al. T… ↗
  2. Preprint at https://doi.org/10.48550/arxiv.2203.02155 (2022). 4. OpenAI. GPT-4 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2023). \#\# 5. Wei, J. et al. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ↗
  3. Preprint at https://doi.org/10.48550/arxiv.2201.11903 (2022). 6. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. Preprint at https://doi.org/10.48550/arxiv.2205.11916 (2022). 7. Huang, J. & Chang, K. C.-C. Towards Reasoning in Large Language M… ↗
  4. Preprint at https://doi.org/10.48550/arxiv.2210.03350 (2022). 11.Jung, J. et al. Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations. Preprint at https://doi.org/10.48550/arxiv.2205.11822 (2022). 12.Dua, D., Gupta, S., Singh, S. & Gardner, M. Successive Prompting for Decom… ↗
  5. Trans. Assoc. Comput. Linguist. 9, 790–806 (2021). 33.Ling, W., Yogatama, D., Dyer, C. & Blunsom, P. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  6. (Association for Computational Linguistics, 2017). doi:10.18653/v1/P17-1015. 34.Miao, S., Liang, C.-C. & Su, K.-Y. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 975–984 (Ass… find
  7. (Association for Computational Linguistics, 2016). doi:10.18653/v1/N16-1136. 37.Patel, A., Bhattamishra, S. & Goyal, N. Are NLP Models really able to Solve Simple Math Word Problems? in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ott-2023-thoughtsource-central-hub-large-language/