What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.3.pdf

Summary

Language-model pre-training clearly helps downstream NLP tasks, but it remains unclear which linguistic abilities models such as BERT actually acquire. Ettinger addresses this with a diagnostic suite adapted from psycholinguistic experiments. The selected experiments are ones where human cloze judgments and N400 brain responses diverge. The tests probe models through word prediction in context, so no fine-tuning or representation probing is needed. They cover areas beyond syntax: commonsense/pragmatic inference (CPRAG-102), semantic roles and event knowledge (ROLE-88), and category membership with negation (NEG-136). Each test measures both prediction accuracy and whether good completions receive higher probability than bad ones.

Applied to BERT BASE and BERT LARGE, the suite yields a mixed picture:

• BERT retrieves hypernyms very well.
• It mostly separates good completions from same-category alternatives and from role reversals, though with much smaller margins than humans.
• It struggles with difficult inferences and role-based event prediction.
• It largely ignores negation.

These findings suggest that a purely predictive training objective may not teach meaning-dependent phenomena like truth reversal.

Key points

• Three diagnostic sets are released with code (github.com/aetting/lm-diagnostics): CPRAG-102, ROLE-88 and NEG-136. NEG-136 has two variants, NEG-136-SIMP and NEG-136-NAT.
• On CPRAG-102, both BERT models place the best completion among their top 5 predictions in roughly half of the items. Perturbation tests show the models depend on word order and full context, but some correct answers seem to exploit shortcuts.
• On ROLE-88, BERT LARGE matches the top human prediction in only 37.5% of items. Both models usually still rank good completions above role reversals.
• On NEG-136-SIMP, BERT favours the true completion in 100% of affirmative sentences but in 0% of negated ones, defaulting to the subject's category.
• With more natural negated sentences (NEG-136-NAT), true statements are preferred in 75% of items for BERT BASE and 87.5% for BERT LARGE. Less natural items reach only 37.5%, a gap the author attributes mainly to training-data frequency.