Elamur Research⌕
☰
home / papers / 2026 / The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts

The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts

Paper · 2026 · Sangmitra Madhusudan, Kaige Chen and Ali Emami

Summary

When a language model correctly answers questions about a sentence with nested relative clauses, it is hard to tell whether it tracked the syntax or simply guessed from familiar real-world associations. The authors build CenterBench to separate the two. It contains 360 center-embedded sentences at nesting depths 1–6. Each plausible sentence (e.g., dogs chasing cats) has a structurally identical implausible twin in which entities perform actions typical of other entities, such as a mailman prescribing medicine. Every sentence comes with six types of comprehension questions, ranging from simple action lookup to multi-step causal tracing. If a model really relied on structure, its accuracy should be the same on both versions.

Across six models, accuracy falls as nesting deepens, and the advantage for plausible sentences grows steadily with depth. This pinpoints where models stop analyzing structure and start leaning on semantics. Humans show no such consistent pattern. The benchmark is therefore a diagnostic for settings where semantic cues may be misleading, such as figurative, historical or adversarial text.

Key points

  • Dataset size and layout: CenterBench has 9,720 questions, split evenly into 4,860 plausible and 4,860 implausible items. They come from 30 sentences per complexity level per subset.
  • How the sentences were built:
    • Plausible sentences were generated with GPT-4-0613 from noun lists based on the Van Overschelde et al. (2004) norms, covering animals, occupations and vehicles.
    • Implausible sentences were made by circular verb swapping, using entity-specific verbs produced with claude-sonnet-4-20250514.
    • All sentences were checked by hand for temporal, semantic and syntactic validity.
  • Question types: There are six, grouped into three tiers:
    • Easy: Action Performed and Agent Identification.
    • Medium: Entity Count and Nested Dependency.
    • Hard: Causal Sequence and Chain Consequence.
  • Main findings:
    • Median gaps between plausible and implausible sentences reach as high as 26.8 percentage points.
    • On questions about resulting actions, plausibility actually lowers accuracy.
  • Reasoning models and humans:
    • Reasoning models score higher, yet inspecting their reasoning reveals that they sometimes fall back on meaning-based shortcuts, reason excessively, or decline to state implausible relations.
    • Humans were tested up to level 4 and showed inconsistent plausibility effects.
    • The data and code are public on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Emory UniversityAtlanta · United States36 works · 39,779 citations🎓Brock UniversitySt. Catharines · Canada3 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Sangmitra Madhusudan, Kaige Chen and Ali Emami. 2026. The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.19

@inproceedings{madhusudan2026dog,
  title = {{The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts}},
  author = {Sangmitra Madhusudan, Kaige Chen and Ali Emami},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.19},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.19}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet. [Accessed 15-07-2025]. ↗
  2. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
  3. Nathaniel Berger, Stefan Riezler, Sebastian Ebert, and Artem Sokolov. 2021. Don’t search for a search method — simple heuristics suffice for adversarial text attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8216–8224, Online and Punta Cana, Do… find
  4. Iona Carslaw, Sivan Milton, Nicolas Navarre, Ciyang Qing, and Wataru Uegaki. 2025. Automatic extraction of clausal embedding based on large-scale english text data. Preprint, arXiv:2506.14064.
  5. Noam Chomsky and George A. Miller. 1963. Introduction to the formal analysis of natural languages. Journal of Symbolic Logic, 33(2):299–300. find
  6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. Deepseek-r1: Incentivizing reasoning capability in l…
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025b. Deepseek-v3 technical report. Preprint, arXiv:… find
  8. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on composition…
  9. Benjamin Estermann, Luca A. Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer. 2024. Puzzles: A benchmark for neural algorithmic reasoning. Preprint, arXiv:2407.00401. — [arXiv:2407.00401](https://arxiv.org/abs/2407.00401) find
  10. Fernanda Ferreira, Karl GD Bailey, and Vittoria Ferraro. 2002. Good-enough representations in language comprehension. Current directions in psychological science, 11(1):11–15. find
  11. Candice Frances. 2024. Good enough processing: what have we learned in the 20 years since ferreira et al.(2002)? Frontiers in Psychology, 15:1323700. find
  12. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for…
  13. E Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1):1–76. find
  14. Daniel Hardt. 2025. Sparks of pure competence in llms: the case of syntactic center embedding in english. Society for Computation in Linguistics, 8(1).
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  16. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, Adriane Boyd, and 1 others. 2020. spacy: Industrial-strength natural language processing in python. find
  17. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  18. Wenyue Hua, Tyler Wong, Fei Sun, Liangming Pan, Adam Jardine, and William Yang Wang. 2025. InductionBench: LLMs fail in the simplest complexity class. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26526–26546, Vienna, Austri…
  19. Richard Hudson. 1996. The difficulty of (so-called) self-embedded structures. Work. Pap. Linguist, 8:283– 314. find
  20. Liwei Kang, Zirui Zhao, David Hsu, and Wee Sun Lee. 2024. On the empirical complexity of reasoning and planning in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2897–2936, Miami, Florida, USA. Association for Computational Linguistics. find
  21. Fred Karlsson. 2007. Constraints on multiple center-embedding of clauses. Journal of Linguistics, 43(2):365–392. find
  22. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. find
  23. Yair Lakretz, Théo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. 2022. Can transformers process recursive nested constructions, like humans? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3226–3232, Gyeongju, Republic of Korea. International Committee on… find
  24. Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699. find
  25. Maria Lapata, Scott McDonald, and Frank Keller. 1999. Determinants of adjective-noun plausibility. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 30–36, Bergen, Norway. Association for Computational Linguistics. find
  26. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Re…
  27. Sangmitra Madhusudan, Robert Morabito, Skye Reid, Nikta Gohari Sadr, and Ali Emami. 2025. Fine-tuned LLMs are “time capsules” for tracking societal bias through books. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human… find
  28. George A. Miller and Noam Chomsky. 1963. Finitary models of language users. In D. Luce, editor, Handbook of Mathematical Psychology, pages 2–419. John Wiley & Sons. find
  29. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. 2024. A glitch in the matrix? locating and detecting language model grounding with fakepedia. In Proceedings of the 62nd Annual Meeting of the Association for… find
  30. John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020. Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3829–3839, Online. Association for Computational Linguistics.
  31. Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024. In-context learning generalizes, but not always robustly: The case of syntax. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1… find
  32. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. Preprint, arXiv:2211.05655.
  33. Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger. 2024. Plausibly problematic questions in multiple-choice benchmarks for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3451–3473, Miami, Florida… find
  34. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, and Mantas Mazeika et al. 2025. Humanity’s last exam. Pr…
  35. Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, and James Glass. 2024. Quantifying generalization complexity for large language models. Preprint, arXiv:2410.01769. — [arXiv:2410.01769](https://arxiv.org/abs/2410.01769) find
  36. Ethan Wilcox, Pranali Vani, and Roger Levy. 2021. A targeted assessment of incremental processing in neural language models and humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… find
  37. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American… find
  38. Linhao Zhang, Jintao Liu, Li Jin, Hao Wang, Kaiwen Wei, and Guangluan Xu. 2024. GOME: Grounding-based metaphor binding with conceptual elaboration for figurative language illustration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18500–18510, Miami… find
  39. Valentin Six, Evan Dufraisse, and Gaël de Chalendar. 2025. Decompositional reasoning for graph retrieval with large language models. Preprint, arXiv:2506.13380. — [arXiv:2506.13380](https://arxiv.org/abs/2506.13380) find
  40. James P Van Overschelde, Katherine A Rawson, and John Dunlosky. 2004. Category norms: An updated and expanded version of the battig and montague (1969) norms. Journal of Memory and Language, 50(3):289–335. find
  41. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics, 7:387– 401.
  42. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Preprint, arXiv:2002.10957. find
  43. Ethan Wilcox, Roger Levy, and Richard Futrell. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 181–190, Florence, Italy. Association f… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/madhusudan-2026-dog-cat-chased-stumped-model/