Elamur Research⌕
☰
home / papers / 2026 / The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

Paper · 2026 · Nathan Roll, Jill Kries, Flora Jin et al.

Summary

Clinical aphasia tests assume a human patient who can see pictures, hear speech, gesture and reflect on their own performance. Text-only LLMs can do none of this, so these tests cannot be applied to them directly. The authors worked with speech-language pathologists to build the Text Aphasia Battery (TAB), a text-only adaptation of the Quick Aphasia Battery (QAB). It keeps the language dimensions that remain visible in transcripts: word retrieval, syntax, discourse coherence and faithful repetition. Its four subtests draw their scoring criteria from QAB and the APROCSA rating system.

To make large-scale use practical, the team tested Gemini 2.5 Flash as an automatic rater against expert annotators. When features are weighted by how often they occur, the model's agreement with the expert consensus is roughly as high as the agreement between two human experts. Baseline runs on small open models show that TAB separates models by task compliance and output quality. The authors present these runs as calibration, not as evidence of "aphasia" in models. The benchmark is meant as the measurement tool for companion work on artificially lesioned models.

Key points

  • Four subtests:
    • Connected Text: open prompts scored on 19 APROCSA-based features.
    • Word Comprehension: forced choice among six options.
    • Sentence Comprehension: yes/no items covering passives, negation and conditionals.
    • Repetition: verbatim reproduction of items that grow longer and more complex.
  • Item counts: each subtest administers five items. The full prompt pool contains about 104 prompts.
  • Rater validation: prevalence-weighted Cohen's κ was 0.255 for Gemini 2.5 Flash vs. consensus and 0.286 for human vs. human. The authors caution that unweighted averaging can mislead when some features are rare.
  • Baseline symptom burden (mean symptoms per response):
    • Llama-3.2-1B: 3.21
    • OLMo-2-1B: 1.98
    • OLMo-2-1B-Instruct: 0.83
    • Gemma-3-1B-it: 0.35
    • Llama-3.2-1B-Instruct: 0.15
  • Release and scope: TAB is released under the MIT License. It is English-only, not intended for clinical diagnosis, and uses AphasiaBank as a human reference resource.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations🎓Temple UniversityPhiladelphia · United States11 works · 24,319 citations

Also on the byline: San Diego State University.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain and Laura Gwilliams. 2026. The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.clpsych-1.27

@inproceedings{roll2026text,
  title = {{The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models}},
  author = {Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain and Laura Gwilliams},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.27},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.27}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across… find
  2. Carolyn Bruce and Anne Edmundson. 2010. Letting the CAT out of the bag: A review of the Comprehensive Aphasia Test. commentary on howard, swinburn, and porter, “putting the CAT out: What the Comprehensive Aphasia Test has to offer”. Aphasiology, 24(1):79–93. find
  3. Marianne Casilio, Kindle Rising, Pélagie M. Beeson, Kate Bunton, and Stephen M. Wilson. 2019. Auditory-perceptual rating of connected speech in aphasia. American Journal of Speech-Language Pathology, 28(2):550–568. find
  4. Heather M. Clark, Rene L. Utianski, Joseph R. Duffy, Edythe A. Strand, Hugo Botha, Keith A. Josephs, and Jennifer L. Whitwell. 2020. Western Aphasia Battery–Revised profiles in primary progressive aphasia and primary progressive apraxia of speech. American Journal of Speech-Language Pathology, 29(1S… find
  5. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. find
  6. Yan Cong, Arianna N. LaCroix, and Jiyeon Lee. 2024. Clinical efficacy of pre-trained large language models through the lens of aphasia. Scientific Reports, 14(1):15573. find
  7. Margaret Forbes, Davida Fromm, and Brian MacWhinney. 2012. Aphasiabank: A resource for clinicians. Seminars in Speech and Language, 33(3):217–222. find
  8. Ajay D. Halai, Blanca De Dios Perez, James D. Stefaniak, and Matthew A. Lambon Ralph. 2022. Efficient and effective assessment of deficits and their neural bases in stroke aphasia. Cortex, 155:333–346. find
  9. Judge Dependence and Robustness Automated evaluation currently relies on a single judge model and prompt. Future work will evaluate robustness across multiple judge models and prompts and include human adjudication of disagreements to bound judge-specific bias. find
  10. Andrew Kertesz. 2022. The Western Aphasia Battery: A systematic review of research and clinical applications. Aphasiology, 36(1):21–50. find
  11. Psychometrics and Item Design Future work will increase subtest difficulty to mitigate ceiling effects (e.g., harder foils, broader syntactic phenomena, balanced Yes/No) and perform item-response analysis to estimate item difficulty and discrimination. We will also analyze repetition errors under a… find
  12. Brian MacWhinney, Davida Fromm, Margaret Forbes, and Audrey Holland. 2011. Aphasiabank: Methods for studying discourse. Aphasiology, 25(11):1286– 1307. find
  13. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. find
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find
  15. Scoring: For each response, identify the presence (1) or absence (0) of APROCSA-based aphasic features (see Section B.5). B.2 find
  16. Nathan Roll, Jill Kries, Laura Gwilliams, and Cory Shain. 2026. Artificial aphasias in lesioned language models. Preprint, arXiv:2605.16222. — [arXiv:2605.16222](https://arxiv.org/abs/2605.16222) find
  17. Katherine Salter, Jeffrey Jutai, Norine Foley, Chelsea Hellings, and Robert Teasell. 2006. Identification of aphasia post stroke: A review of screening assessment tools. Brain Injury, 20(6):559–568. find
  18. Subtest 2: Word Comprehension Objective: Evaluates lexical-semantic processing and selection among competing meanings. Instructions: Elicit responses verbatim. Provide the options alongside each question. Require a one-token answer corresponding exactly to one of the listed options; do not include e… find
  19. Luise Springer and Stefanie Mantey. 2010. The Comprehensive Aphasia Test: A review. commentary on howard, swinburn, and porter, “putting the CAT out: What the Comprehensive Aphasia Test has to offer”. Aphasiology, 24(1):75–78. 1. “Which one is an animal with a mane? (lion, drum, violin, giraffe, boo…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roll-2026-text-aphasia-battery-tab-clinically/