Deep Biaffine Attention for Neural Dependency Parsing

What it is about

The paper builds on the neural graph-based parser of Kiperwasser and Goldberg. It uses a larger and more heavily regularised BiLSTM encoder, passes the top recurrent states through dimension-reducing MLPs, and replaces the MLP attention and the affine label classifier with biaffine ones: one for arc scores over all head-dependent pairs and one for labels. Each arc and label score is a biaffine function of a head-side and a dependent-side vector, which keeps the classifier close in form to an affine one. On the English Penn Treebank the parser reaches 95.7 UAS and 94.1 LAS, which is 1.8 and 2.2 points above Kiperwasser and Goldberg and close to the best transition-based parser of the time (95.8 UAS). The paper also reports which hyperparameter choices change accuracy most.

Ideas

I1. gparse: check that the arc-head and arc-dependent representations, and the label-head and label-dependent representations, each pass through their own dimension-reducing MLP before the biaffine scorer, and add them if not; measure LAS on EWT dev (cv).
I2. gparse: report UAS and LAS in the same run with separate loss sums for the arc and label heads, and check whether the label loss hurts the arc score on EWT dev, as the paper's joint training assumes it does not.
I3. gparse: run the paper's hyperparameter sweep on dropout and the regularisers only (a small grid on EWT dev) before any architectural change, since its results say which settings move accuracy most; start only after the operator's go-ahead, because each GPU run is expensive.

For Mova

The ideas touch the biaffine heads in gparse and its training loop, and they use EWT dev through the cv folds. The chapter that matters most is languages/en/20-syntax-overview, with languages/en/23-complementation for the label set. Measure first I2, which is cheap and tells whether the joint loss costs arc accuracy before any change to the architecture.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1611.01734

Summary

Neural graph-based dependency parsers were simpler than transition-based ones but less accurate. Dozat and Manning extend the BiLSTM graph-based parser of Kiperwasser & Goldberg (2016) to narrow this gap. Each word's top BiLSTM state first goes through small ReLU MLPs. These compress it into separate "dependent" and "head" views. A biaffine transformation then scores every possible arc, and a second biaffine classifier assigns relation labels. The biaffine form models two things: how readily a word accepts dependents at all, and how well a specific head–dependent pair fits. The network is bigger than earlier ones but uses dropout at every stage. At test time, an MST algorithm ensures the output is a valid tree. The evaluation covers six languages: English PTB converted to Stanford Dependencies (3.3.0 and 3.5.0), the Chinese Penn Treebank, and CoNLL 09 data. The authors also run ablations on architecture and hyperparameters. The result is the strongest graph-based parser on PTB and close to the best transition-based system, while keeping a simple design. This makes it a strong baseline for later parsers.

Key points

On PTB-SD 3.3.0 the parser reaches 95.7% UAS and 94.1% LAS. This is 1.8 and 2.2 points better than Kiperwasser & Goldberg (2016) and near Kuncoro et al. (2016), which scores 95.8% UAS and 94.6% LAS.
Configuration: 100-dim uncased word and POS embeddings, 3 BiLSTM layers (400 units per direction), and arc/label MLPs of 500/100 dims. Dropout is 33% throughout, and training uses annealed Adam for roughly 50,000 steps.
The deep biaffine classifier matched the shallow variant on unlabeled accuracy but ran faster and overfit less. The shallow label classifier is much larger (801×c×801 vs 101×c×101).
Hyperparameters were tuned on PTB-SD 3.5.0 validation data to avoid overfitting the more popular 3.3.0 benchmark.
The remaining gap is mainly in LAS. The authors suggest possible causes: GloVe embeddings or POS-tagger errors, weaker modeling of phrase composition, or general limits of graph-based parsing.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1611.01734

@misc{dozat2016deep,
  title = {{Deep Biaffine Attention for Neural Dependency Parsing}},
  author = {Dozat, Timothy and Manning, Christopher D.},
  year = {2016},
  eprint = {1611.01734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.01734}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. In Association for Computational Linguistics, 2016. URL https://arxiv.org/abs/1603.06042. ↗ our card
Gabor Angeli, Melvin Johnson Premkumar, and Christopher D Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), 2015. find our card
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2014. find our card
Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A Smith. Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing, 2016. find
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. ACL 2016, 2016. find
Danqi Chen and Christopher D Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pp. 740–750, 2014. find our card
Hao Cheng, Hao Fang, Xiaodong He, Jianfeng Gao, and Li Deng. Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076, 2016. find
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. Transition-based dependency parsing with stack long short-term memory. Proceedings of the conference on empirical methods in natural language processing, 2015. find our card
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2015. find
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 2015. find
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587, 2016. find our card
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014. find
Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327, 2016. find our card
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774, 2016. URL http://arxiv.org/abs/1611.05774. ↗
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL 2014, pp. 302–308, 2014. find
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. Empirical Methods in Natural Language Processing, 2015. find
Ankur P Parikh, Hoifung Poon, and Kristina Toutanova. Grounded semantic parsing for complex knowledge extraction. In Proceedings of North American Chapter of the Association for Computational Linguistics, pp. 756–766, 2015. find
Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp… find
Kristina Toutanova, Xi Victoria Lin, and Wen-tau Yih. Compositional learning of embeddings for relation paths in knowledge bases and text. In ACL, 2016. find
David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. Structured training for neural network transition-based parsing. Annual Meeting of the Association for Computational Linguistics, 2015. find