Assessing BERT's Syntactic Abilities

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1901.05287

Summary

This paper tests whether BERT handles English syntactic dependencies that earlier studies had probed in LSTMs. BERT is a Transformer built only on self-attention, with absolute position embeddings and no recurrent state, so it was not obvious that it would. Goldberg reuses three existing stimulus sets:

• naturally occurring Wikipedia agreement sentences from Linzen et al. (2016);
• nonce sentences from Gulordava et al. (2018), where content words are swapped for random words with the same part of speech and inflection, so the model can't rely on meaning;
• controlled minimal pairs from Marvin and Linzen (2018) for subject-verb agreement and reflexive anaphora.

Because BERT reads context in both directions, the focus verb is replaced with [MASK] and the rest of the sentence stays visible. The model counts as correct when it scores the right verb form above the wrong one. The pre-trained bert-base-uncased and bert-large-uncased models are used with no fine-tuning. Both get high accuracy on every test and usually beat the published LSTM numbers. The setups differ, so this comparison is only a rough guide. Still, the result goes against the expectation that attention-only models struggle with hierarchical structure, and it helped prompt later probing of syntax in pretrained Transformers.

Key points

• Linzen et al. (2016) sentences: accuracy stays at 0.96–0.97 for both models with one to four agreement attractors. There are 24,031 sentences with one attractor and 254 with four.

• Gulordava et al. (2018) nonce set (383 pairs): BERT Base scores 0.83 and BERT Large 0.80. The LSTM result reported in the original work is 74.1 ± 1.6.

• Marvin and Linzen (2018) pairs, where BERT wins: BERT is far ahead on long-distance agreement. For example, across an object relative clause BERT Base scores 0.89 and the LSTM 0.50.

• Marvin and Linzen (2018) pairs, where the LSTM wins: the LSTM is better only on sentential complements and short VP coordination. Goldberg attributes this to the LSTM overfitting those constructions. BERT Base often outperforms BERT Large.

• Discarded stimuli: three kinds were removed.

  • Negative polarity item (NPI) pairs, because they differ in two words.
  • is/are cases in the naturalistic sets, because copular sentences give extra agreement cues.
  • Verbs that are not single wordpieces in BERT's vocabulary. This removed 23,368 of 152,300 Marvin and Linzen pairs, 680 Linzen sentences and 28 Gulordava pairs.

  Code is at github.com/yoavg/bert-syntax.