An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)
Paper · 2023 · Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu et al.
Summary
The paper investigates how ChatGPT handles math word problems (MWPs) and which properties of a problem predict that the model will get it wrong. The authors describe this as the first independent study of the model. They submitted all 1,000 problems from DRAW-1K, which pairs each problem with answers and template algebraic equations, through an unofficial CLI Python wrapper, since no API existed then. They ran three settings: January 2023 with an answer-only prompt, February 2023 with a revised answer-only prompt, and February 2023 with ChatGPT free to explain its reasoning.
Error rates depended heavily on whether the model showed its work. It failed on about 84% of problems with answer-only output and about 20% when it wrote out its solution. Counts of unknowns and arithmetic operations, taken from the gold equations, correlated with failure. The released response dataset and baseline classifiers that predict failure support work on introspective front-ends that judge in advance whether an LLM can be trusted on a given input.
Key points
- Answer-only runs: in January, 831 answers were wholly incorrect and 9 claimed "No Solution", giving 84% failure. February results were almost identical.
- Runs with visible work: 513 problems were fully solved (51%), and counting partially correct outputs raises this to about 80%. For comparison, the EPT model reports 59% accuracy.
- Addition/subtraction counts: failure probability grows linearly with these in every setting, with R² of 0.821 in January, 0.870 in February without work, and 0.915 with work shown.
- Multiplication/division counts:
- These show a linear link to failure only when work is shown (R² = 0.802).
- The number of unknowns stops mattering in that setting.
- Failure-prediction baselines:
- Random forest (scikit-learn) and XGBoost classifiers were evaluated with five-fold cross-validation.
- Their features come from oracle ground-truth equations.
- Precision on incorrect answers reaches 0.90–0.98 in the answer-only settings.
- The response data is published at github.com/lab-v2/ChatGPT_MWP_eval.
Institutions
United StatesFields, topics and concepts
📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksMathematics · 189 worksApplied Mathematics · 52 works
🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 works
Cite
Cite as Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu. 2023. An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP). arXiv:2302.13814 · AAAI Spring Symposium 2023 (MAKE).
@misc{shakarian2023independent,
title = {{An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)}},
author = {Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu},
year = {2023},
doi = {10.48550/arxiv.2302.13814},
eprint = {2302.13814},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2302.13814}
}Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/shakarian-2023-independent-evaluation-chatgpt-mathematical-word/