Elamur Research⌕
☰
home / papers / 2025 / From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

Paper · 2025 · Luoxi Tang, Tharunya Sundar, Yuqiao Meng et al.

Summary

Benchmarks that test LLMs on standardized English exams usually check only whether the final answer is right. That says little about whether a model could work as a tutor, which needs sound intermediate reasoning and the ability to explain why wrong options are tempting. The authors treat solving each exam item as a path through named cognitive sub-skills, an idea drawn from cognitive diagnosis models and evidence-centered design. They release ESTBOOK, which has 10,576 objective questions of 29 types from TOEFL, IELTS, SAT, GRE and GMAT. The items span text, maths notation, images, tables and audio. Each question carries a step-by-step trajectory and rationales describing the trap behind every distractor.

Scoring the steps separately shows a clear pattern. Models handle the opening steps well but break down on later multi-step integration. They are also easily persuaded by well-crafted wrong options. Prompts that force the model through the weak step give consistent gains. The work matters because it turns a single accuracy number into a diagnosis of where reasoning fails.

Key points

  • ESTBOOK has 10,576 questions in 29 task types from five exams (SAT, GRE, GMAT, TOEFL, IELTS), sourced from official prep guides and vetted open materials. Proprietary visuals were rebuilt by hand.
  • Only objective formats are used (multiple choice, deterministic numeric entry), so that intermediate steps can be scored without ambiguity.
  • Tasks are grouped by how much they rely on knowledge versus reasoning, with a third integrative group. Each sub-type has a three-node path:
    • structural: syntax_parse → rule_match → predict;
    • semantic: semantic_parse → logical_scope → resolve;
    • data interpretation: analytical_goal → visual_parse → analyze.
  • First-step accuracy reaches up to 97%, but later steps are weaker and unstable. Reasoning complexity hurts models more than input length does.
  • Bottleneck-targeted prompts beat plain CoT for GPT-4V, GPT-5 and Claude-S-4. The largest effect is on GMAT-IR with table alignment, which rose from roughly 14–16 to roughly 60–72.

Interesting ideas

  • I1. Break every task type into a fixed, short chain of named sub-skills and score each link, so a failure can be traced to one node instead of the whole item.
  • I2. Annotate each wrong option with the specific misconception it exploits, then test whether the system explicitly rejects it. This kind of discriminative robustness turned out to be separate from simply generating the correct answer.
  • I3. Distractors built on partial semantic truths were the hardest. Near-miss options are therefore a sharper probe of meaning than random wrong ones.
  • I4. Force syntax first: make the solver name grammatical roles such as contrast markers before judging meaning, so that plausibility cannot override structural validity.
  • I5. Require an explicit text-to-table alignment step before any arithmetic. This gave the largest gain in the paper.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test readers could log which stage (parse, frame, world lookup, computation) produced the wrong answer, turning test scores into a per-stage error map.
  • I2 (learn): Evaluation sets could include distractor answers with labelled traps as negative controls, checking that the world model actively refutes them rather than merely preferring the gold answer.
  • I3 (frames): Generate near-miss paraphrases that swap a role, scope or negation, and verify that frame extraction tells them apart from the original.
  • I4 (tree): Sentence-completion answers could be accepted only when the candidate yields a valid UD attachment and the expected relation at the connective, using the parse as a hard gate.
  • I5 (world): Word-problem solving could first bind every quantity mentioned in the text to a world entity or table cell, and compute only after that binding is complete and inspectable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dartmouth CollegeHanover · United States37 works · 42,625 citations🎓Binghamton UniversityBinghamton · United States3 works · 424 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Luoxi Tang, Tharunya Sundar, Yuqiao Meng, Shuai Yang, Ankita Patra, Lakshmi Manohar Chippada, Jiqian Zhao, Yi Li, Weicheng Ma and Zhaohan Xi. 2025. From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests. arXiv:2505.17056.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.17056 · arXiv 2505.17056

@misc{tang2025test,
  title = {{From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests}},
  author = {Luoxi Tang, Tharunya Sundar, Yuqiao Meng, Shuai Yang, Ankita Patra, Lakshmi Manohar Chippada, Jiqian Zhao, Yi Li, Weicheng Ma and Zhaohan Xi},
  year = {2025},
  doi = {10.48550/arxiv.2505.17056},
  eprint = {2505.17056},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.17056}
}

References

67 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramatu Oiza Abdulsalam and Segun Aroyehun. 2025. Large language models approach expert pedagogical quality in math tutoring but differ in instructional and linguistic profiles. arXiv preprint arXiv:2512.20780. ↗
  2. GMAT Club. 2025. Gmat club: Best gmat prep, tests, mba admissions and courses. find
  3. Robert Godwin-Jones. 2024. Distributed agency in second language learning and teaching through generative ai. arXiv preprint arXiv:2403.20216. ↗
  4. Allison Andreyev. 2025. Quantization for openai’s whisper models: A comparative analysis. arXiv preprint arXiv:2503.09905. ↗
  5. Graduate Management Admission Council (GMAC). 2025. GMAT Official Guide 2025–2026 Bundle: Books + Online Question Bank, 1st edition. Wiley. find
  6. Jed I Appelrouth and Karen M Zabrucky. 2017. Preparing for the sat: A review. College and University, 92(1):2. find
  7. Calbert Graham and Nathan Roll. 2024. Evaluating openai’s whisper asr: Performance analysis across diverse accents and speaker traits. JASA Express Letters, 4(2). ↗
  8. Vahid Ashrafimoghari, Necdet Gürkan, and Jordan W Suchow. 2024. Evaluating large language models on the gmat: Implications for the future of business education. arXiv preprint arXiv:2401.02985. ↗
  9. Nursultan Askarbekuly and Nenad Aničić. 2024. Llm examiner: automating assessment in informal self-directed e-learning using chatgpt. Knowledge and Information Systems, 66(10):6133–6150. ↗
  10. Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. 2025. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. Preprint, arXiv:2412.09078. ↗
  11. College Board. 2025. The cognitively complex thinking required by select sat suite questions: Evidence from students with specific learning disorders affecting reading (dyslexia). Technical report, College Board. Cognitive Lab Report. find
  12. Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, and 1 others. 2025. Toward generalizable evaluation in the llm era: A survey beyond benchmarks. arXiv preprint arXiv:2504.18838. ↗
  13. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.
  14. Cheng-Han Chiang, Wei-Chih Chen, Chun-Yi Kuan, Chienchou Yang, and Hung-yi Lee. 2024. Large language model as an assignment evaluator: Insights, feedback, and challenges in a 1000+ student course. arXiv preprint arXiv:2407.05216. ↗
  15. College Board. 2022. Digital sat sample questions and answer explanations. https://satsuite.collegeboard.org/media/ pdf/digital-sat-sample-questions.pdf. ↗
  16. Yang Feng and Xiya Wang. 2024. Exploring the development of chinese college students’ proficiency in english through chatgpt: An experimental study. In Proceedings of the 2024 the 16th International Conference on Education Technology and Computers, pages 148–154. find
  17. Gruber’s Complete GRE Guide 2012. Sourcebooks, Inc. find
  18. Alex Guilherme. 2019. Ai and education: the importance of teacher and student relations. AI & society, 34:47–54. find
  19. Pranav Gupta. 2023. Testing llm performance on the physics gre: some observations. arXiv preprint arXiv:2312.04613. ↗
  20. Lisa Zimmer Hatch, Scott A. Hatch, and Sandra Luna McCune. 2023. GMAT Prep 2024/2025 For Dummies (GMAT Focus Edition): Book + 3 Practice Tests + 100 Flashcards Online, 11th edition. For Dummies. find
  21. Wayne Holmes and Ilkka Tuomi. 2022. State of the art and practice in ai in education. European journal of education, 57(4):542–570. ↗
  22. Danial Hooshyar, Yeongwook Yang, Gustav Šíř, Tommi Kärkkäinen, Raija Hämäläinen, Mutlu Cukurova, and Roger Azevedo. 2025. Problems with large language models for learner modelling: Why llms alone fall short for responsible tutoring in k–12 education. arXiv preprint arXiv:2512.23036. ↗
  23. IELTS-up. 2023. Ielts listening practice tests. https://ielts-up.com/listening/ ielts-listening-practice.html. ↗
  24. Christopher J. Johnstone, Nicole A. Bottsford-Miller, and Sandra J. Thompson. 2006. Using the think aloud method (cognitive labs) to evaluate test design for students with disabilities and english language learners. Technical Report 44, University of Minnesota, National Center on Educational Outcomes. find
  25. Anghelo Josué, Mirna Carolina Bedoya-Flores, Erick Fabián Mosquera-Quiñonez, Ángel Enrique Mesías-Simisterra, and José Vicencio Bautista-Sánchez. 2023. Educational platforms: Digital tools for the teaching-learning process in education. Ibero-American Journal of Education & Society Research, 3(1):259–263. find
  26. Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2024. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21258–21266. find
  27. Yuqiao Meng, Luoxi Tang, Dazheng Zhang, Rafael Brens, Elvys J Romero, Nancy Guo, Safa Elkefi, and Zhaohan Xi. 2026. Small agent group is the future of digital health. arXiv preprint arXiv:2602.08013. ↗
  28. Unggi Lee, Sanghyeok Lee, Junbo Koh, Yeil Jeong, Haewon Jung, Gyuri Byun, Yunseo Lee, Jewoong Moon, Jieun Lim, and Hyeoncheol Kim. 2023. Generative agent for teacher training: Designing educational problem-solving simulations with large language model-based agents for pre-service teachers. NeurIPS’23 Workshop on Generative AI for Education (GAIED). find
  29. Robert J Mislevy, Russell G Almond, and Janice F Lukas. 2003. A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1):i–29. ↗
  30. Jacqueline Leighton and Mark Gierl. 2007. Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press. find
  31. Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:2410.05318. ↗
  32. Naiming Liu, Shashank Sonkar, and Richard Baraniuk. 2025. Do llms make mistakes like students? exploring natural alignments between language models and human error patterns. In International Conference on Artificial Intelligence in Education, pages 364–377. Springer. find
  33. Eric Loken, Filip Radlinski, Vincent H. Crespi, Josh Millet, and Lesleigh Cushing. 2004. Online study behavior of 100,000 students preparing for the sat, act, and gre. Journal of Educational Computing Research, 30(3):255–262. ↗
  34. Jieyi Long. 2023. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291. ↗
  35. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. ↗
  36. Abdullah Mallik. 2025. Sat suite question bank categories. https://sat-questions.onrender.com/ categories. ↗
  37. Patrick E McKnight and Julius Najab. 2010. Mann-whitney u test. The Corsini encyclopedia of psychology, pages 1–1. ↗
  38. Joshua Mitton, Prarthana Bhattacharyya, Digory Smith, Thomas Christie, Ralph Abboud, and Simon Woodhead. 2026. Misconception diagnosis from student-tutor dialogue: Generate, retrieve, rerank. arXiv preprint arXiv:2602.02414. ↗
  39. Ahmadreza Mohebbi. 2025. Enabling learner independence and self-regulation in language education using ai tools: a systematic review. Cogent Education, 12(1):2433814. ↗
  40. Vrushket More, Lyra Lu, Zeyu Ding, Zhaohan Xi, Seth Mizia, and Nancy L Guo. 2026. Theramind: a multi-llm ensemble for accelerating drug repurposing in lung cancer via case report mining. npj Precision Oncology. ↗
  41. Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. 2024. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.00147. ↗
  42. Juanan Pereira, Juan-Miguel López, Xabier Garmendia, and Maider Azanza. 2024. Leveraging open source llms for software engineering education and training. In 2024 36th International Conference on Software Engineering Education and Training (CSEE&T), pages 1–10. IEEE. find
  43. Hamoun Sabri, Muhammad HA Saleh, Parham Hazrati, Keith Merchant, Jonathan Misch, Purnima S Kumar, Hom-Lay Wang, and Shayan Barootchi. 2025. Performance of three artificial intelligence (ai)-based large language models in standardized testing; implications for ai-assisted dental education. Journal of periodontal research, 60(2):121–133. ↗
  44. SAT Questions. 2023. Sat practice questions and categories. https://sat-questions.onrender.com/ categories. ↗
  45. Maximilian Schmidhuber and Udo Kruschwitz. 2024. Llm-based synthetic datasets: Applications and limitations in toxicity detection. LREC-COLING, 37:2024. find
  46. Yuqiao Meng, Luoxi Tang, Feiyang Yu, Jinyuan Jia, Guanhua Yan, Ping Yang, and Zhaohan Xi. 2025a. Uncovering vulnerabilities of llm-assisted cyber threat intelligence. arXiv preprint arXiv:2509.23573. ↗
  47. Robin Schmucker, Meng Xia, Amos Azaria, and Tom Mitchell. 2024. Ruffle&riley: Insights from designing and evaluating a large language model-based conversational tutoring system. In International Conference on Artificial Intelligence in Education, pages 75–90. Springer. find
  48. Yuqiao Meng, Luoxi Tang, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, and Zhaohan Xi. 2025b. Benchmarking llm-assisted blue teaming via standardized threat hunting. arXiv preprint arXiv:2509.23571. ↗
  49. Luoxi Tang, Yuqiao Meng, Ankita Patra, Weicheng Ma, Muchao Ye, and Zhaohan Xi. 2025. Polar: Automating cyber threat prioritization through llm-powered assessment. arXiv preprint arXiv:2510.01552. ↗
  50. Thanayut Tiratatri, Kanin Sukittivarapunt, Thammathat Sarasinpitak, and Aung Pyae. 2025. Designing an llm-based ielts question generator, assessment, and personalized training system: Architecture and research agenda. In 2025 22nd International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pages 1–6. IEEE. find
  51. TOEFL Test Prep. 2023. Toefl test preparation website. https://toefltestprep.com/. ↗
  52. Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, and Jindong Gu. 2024. Stop reasoning! when multimodal llm with chain-of-thought reasoning meets adversarial image. arXiv preprint arXiv:2402.14899. ↗
  53. Bingbing Wen, Bill Howe, and Lucy Lu Wang. 2024a. Characterizing llm abstention behavior in science qa with context perturbations. arXiv preprint arXiv:2404.12452. ↗
  54. Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. 2024b. Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6743–6744. find
  55. Ron Woldoff. 2024. GRE 5-Hour Quick Prep For Dummies. John Wiley & Sons. find
  56. Ron Woldoff and Joseph Kraynak. 2015. GRE For Dummies: With Online Practice Tests. John Wiley & Sons. find
  57. Eamon Worden, Cristina Heffernan, Neil Heffernan, and Shashank Sonkar. 2026. Foundationalassist: An educational dataset for foundational knowledge tracing and pedagogical grounding of llms. arXiv preprint arXiv:2602.00070. ↗
  58. Wei Xia, Shaoguang Mao, and Chanjing Zheng. 2024. Empirical study of large language models as automated essay scoring tools in english composition_taking toefl independent writing task for example. arXiv preprint arXiv:2401.03401. ↗
  59. Wenbo Xu, Muhammad Shahreeza, Wai Lam Hoo, and Wudao Yang. 2025. Explainable ai for education: Enhancing essay scoring via rubric-aligned chain-of-thought prompting. Preprints. ↗
  60. Shuai Yang, Qi Yang, Luoxi Tang, Yuqiao Meng, Nancy Guo, Jeremy Blackburn, and Zhaohan Xi. 2025. On the eligibility of llms for counterfactual reasoning: A decompositional study. arXiv preprint arXiv:2505.11839. ↗
  61. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  62. Dharunish Yugeswardeenoo, Kevin Zhu, and Sean O’Brien. 2024. Question-analysis prompting improves llm performance in reasoning tasks. arXiv preprint arXiv:2407.03624. ↗
  63. Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, and 1 others. 2024a. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884. ↗
  64. Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024b. Chain of preference optimization: Improving chain-of-thought reasoning in llms. Advances in Neural Information Processing Systems, 37:333–356. find
  65. Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. 2024c. Can llm graph reasoning generalize beyond pattern memorization? arXiv preprint arXiv:2406.15992. ↗
  66. Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, and 1 others. 2024d. Simulating classroom education with llm-empowered agents. arXiv preprint arXiv:2406.19226. ↗
  67. Yang Zhong, Jiangang Hao, Michael Fauss, Chen Li, and Yuan Wang. 2024. Evaluating ai-generated essays with gre analytical writing assessment. arXiv preprint arXiv:2410.17439. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2025-test-taking-cognitive-scaffolding-pedagogical/