📄 Deep RNNs Encode Soft Hierarchical Syntax

Summary

This short paper asks whether deep recurrent networks trained on ordinary NLP objectives learn syntactic structure that they were never directly taught. The authors probe four pre-trained multi-layer LSTM models: a Stanford dependency parser (English Web Treebank, UD), DeepSRL (CoNLL-2012), an OpenNMT encoder (WMT-14 English-German), and forward/backward language models (CoNLL-2012). From each layer's word vectors, a small feed-forward classifier predicts four labels: the POS tag and the constituent labels one, two, and three levels above the word. A second probe tests whether a pair of word vectors indicates a dependency arc in UD.

All four kinds of supervision produce representations that beat per-word majority baselines. The only exception is POS for the parser, which already receives gold tags as input. The layers also form a soft hierarchy: tasks higher in the tree reach peak accuracy in deeper layers. The results suggest that deep RNNs build up abstract syntax internally. This may explain why they work without hand-made linguistic features, and why ELMo-style layer mixing helps.

Key points

• The probe is a single 300-unit ReLU hidden layer with softmax. It is trained on the CoNLL-2012 development set and tested on its test set, so it never sees the RNNs' training data.
• In 11 of 16 model–task combinations, accuracy rises up to some layer and then falls. For SRL, parent-label accuracy is highest at the layer just above where POS is best, and the grandparent and great-grandparent labels reach their best one layer higher again.
• The language model matches the dependency parser on the constituent tasks at layer 2. After that layer, the language model declines while the parser keeps improving.
• On dependency arc prediction, the parser's top layer reaches about 95%. The SRL, MT, and LM models peak roughly 12–20 points above their input layers, while GloVe stays at chance (0.50).
• Baseline task scores of the probed models: parser UAS 91.5 / LAS 82.18; MT BLEU 21.37; LM perplexity 50.56 (forward) and 51.24 (backward) on CoNLL-2012.