Elamur Research⌕
☰
home / papers / 2026 / Anthropocentric Bias in Language Model Evaluation

Anthropocentric Bias in Language Model Evaluation

Paper · 2026 · Raphaël Millière and Charles Rathkopf

Summary

Debates about what large language models (LLMs) can cognitively do usually guard against anthropomorphism but neglect the opposite error: judging LLMs by human standards without justification. This conceptual squib introduces no new dataset or benchmark. The authors observe that researchers often treat poor LLM performance as evidence of missing competence, yet they rarely accept the reverse inference. They read this asymmetry as a sign of anthropocentric bias. To address it, they restate the competence/performance distinction in non-anthropomorphic terms and propose a taxonomy with two neglected forms of bias:

  • Auxiliary oversight: failures caused by factors outside the tested ability are mistaken for incompetence.
  • Mechanistic chauvinism: LLM solutions are rejected only because their internal strategies differ from human ones.

Using published case studies on syntax and analogical reasoning, they show how mismatched test conditions can distort comparisons between humans and models. Their remedy is an empirical, iterative program that combines behavioral experiments with mechanistic interpretability, focused on Marr's computational and algorithmic levels. This matters for anyone designing fair LLM evaluations, including linguistic benchmarks.

Key points

  • Hu and Frank (2024): Directly comparing log probabilities of minimal pairs (e.g., subject–verb agreement) often gave better results than prompting models for explicit grammaticality judgments. The authors treat metalinguistic judgment as an auxiliary task demand.
  • Lampinen (2024), revisiting Lakretz et al. (2022): Human participants received instructions, training and feedback, while models were tested zero-shot. When given a few examples, LLMs matched or exceeded humans on nested agreement, including deeper nesting than humans were tested on.
  • Analogical reasoning debate: Webb, Holyoak, and Lu (2023) reported strong LLM results on letter-string analogies. Lewis and Mitchell (2024) found that performance dropped with permuted alphabets. Webb et al. (2024) replied that the permuted variant adds a letter-counting demand, which is a non-trivial task for LLMs.
  • Human cognition as a starting point: Human capacities should serve as initial search templates (Boyle's "investigative kinds"), not as necessary conditions. Defining competence at the implementational level, for example by requiring a biological substrate, would make the question non-empirical.
  • Limits of interpretability: As work by Makelov and colleagues (2024) and by Huang and colleagues (2023) shows, a network may encode decodable features that it does not causally rely on. Mechanistic studies therefore carry their own risk of anthropomorphic projection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🔬Forschungszentrum JülichJülich · Germany5 works · 1,301 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksHistory and Philosophy of Science · 62 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Raphaël Millière and Charles Rathkopf. 2026. Anthropocentric Bias in Language Model Evaluation. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.582

@article{milliere2026anthropocentric,
  title = {{Anthropocentric Bias in Language Model Evaluation}},
  author = {Raphaël Millière and Charles Rathkopf},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.582},
  url = {https://doi.org/10.1162/coli.a.582}
}

References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bender, Emily M. 2022. On NYT Magazine on AI: Resist the urge to be impressed. Medium. https://medium.com /@emilymenonbender/on-nyt-magazine -on-ai-resist-the-urge-to-be -impressed-3d92fd9a0edd Boyle, Alexandria. 2024. Disagreement & classification in comparative cognitive science. Noûs, n/a(n/a).… — [doi:10.1111/nous.12480](https://doi.org/10.1111/nous.12480) ↗
  2. The New York Times. Dutta, Subhabrata, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. 2024. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. Transactions on Machine Learning Research. https://doi.org/10.48550/arXiv.2402.18312, https://openreview.net… ↗
  3. Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35. https://doi.org/10.48550/arXiv .2205.11916 Lakretz, Yair, Théo Desbordes, Dieuwke Hupkes, and Sta… ↗
  4. Lampinen, Andrew Kyle. 2024. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans. Computational Linguistics, 50(4):1441–1476. https://doi.org/10.1162/coli_a_00525 ↗
  5. Lewis, Martha and Melanie Mitchell. 2024. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv:2402.08955. https://doi.org/10.48550/arXiv .2402.08955 Marr, D. 1982. Vision: A Computational Investigation into the Human Representation and Proces… — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗
  6. W. H. Freeman and Company. Makelov, Aleksandar, Georg Lange, Atticus Geiger, and Neel Nanda. 2024. Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. In the Twelfth International Conference on Learning Representations, ICLR 2024. find
  7. Merrill, William and Ashish Sabharwal. 2024. The expressive power of transformers with chain of thought. In the Twelfth International Conference on Learning Representations, ICLR 2024. https://doi.org/10.48550/arXiv.2310.07923 Millière, Raphaël. 2026. Language models as models of language. In Rya… ↗
  8. Sober, Elliott. 2005. Comparative psychology meets evolutionary biology. In Lorraine Datson and Gregg Mitman, editors, Thinking with Animals: New Perspectives on Anthropomorphism. Columbia University Press, pages 85–99. find
  9. Webb, Taylor, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541. https://doi.org/10.1038/s41562-023 -01659-w, PubMed: 37524930 Webb, Taylor, Keith J. Holyoak, and Hongjing Lu. 2024. Evidence from counterfactual tas… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/milliere-2026-anthropocentric-bias-language-model-evaluation/