Elamur Research
☰
home / papers / 2019 / Simple BERT Models for Relation Extraction and Semantic Role Labeling

Simple BERT Models for Relation Extraction and Semantic Role Labeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.05255

Summary

Shi and Lin examine whether pretrained BERT representations can replace the hand-engineered linguistic inputs, such as POS tags, dependency parses and decoding constraints, that leading relation extraction and semantic role labeling (SRL) systems relied on. Such features depend on parsers, which may not exist for a given language or may break down on out-of-domain text.

For relation extraction, the sentence is paired with its subject and object, and entity mentions are replaced by typed masks. BERT outputs are combined with relative-position embeddings and passed through a BiLSTM and an MLP.

For SRL, the predicate is appended to the input. A predicate-indicator embedding, a BiLSTM and an MLP tag BIO argument labels, and a separate MLP labeler handles predicate sense disambiguation.

Experiments on TACRED, CoNLL 2005, CoNLL 2009 and CoNLL 2012 show these feature-free models matching or beating prior single-model results, and on some benchmarks they also beat ensembles. The work establishes strong, simple baselines and suggests that language pretraining can reduce reliance on syntactic tooling. This is relevant for languages with weaker parsing resources.

Key points

  • TACRED: BERT-LSTM-base reaches F1 67.8 (P 73.3, R 63.1). This is the best reported single-model score. It remains slightly below the Zhang et al. (2018) ensemble (68.2) because of lower recall.
  • CoNLL 2009 predicate sense disambiguation: a plain BERT-base MLP scores 96.32 on dev, 96.88 on test and 90.63 on Brown, beating the feature-based He et al. (2018b).
  • CoNLL 2009 dependency-based SRL (end-to-end): BERT-LSTM-large gets F1 92.4 in-domain and 85.7 on Brown. The previous best, Li et al. (2019), had 90.4 and 81.5, and existing ensembles are also surpassed.
  • Span-based SRL: BERT-LSTM-large scores F1 88.8 on CoNLL 2005, 82.0 on Brown and 86.5 on CoNLL 2012. It outperforms the Ouchi et al. (2018) ensemble except on CoNLL 2012 (87.0), where constrained decoding gives that ensemble higher precision.
  • Setup: the models use BERT base-cased and large-cased, an LSTM hidden size of 768, an MLP hidden size of 300 and a learning rate of 5×10⁻⁵. No syntactic features or declarative constraints are used.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 165 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.05255 · arXiv 1904.05255

@misc{shi2019simple,
  title = {{Simple BERT Models for Relation Extraction and Semantic Role Labeling}},
  author = {Shi, Peng and Lin, Jimmy},
  year = {2019},
  doi = {10.48550/arxiv.1904.05255},
  eprint = {1904.05255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.05255}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Improving relation extraction by pre-trained language representations. In AKBC. find
  2. Xavier Carreras and Lluı́s Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. ↗
  4. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. find
  5. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–18. find our card
  6. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018a. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364–369. find our card
  7. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473–483. find our card
  8. Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018b. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2061–2071. find
  9. Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations. find our card
  10. Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401–2411. find
  11. Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. find
  12. Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 411–420. find our card
  13. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506–1515. find our card
  14. Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1630–1642. find
  15. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237. find
  16. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143– 152. find
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  18. Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1192–1202. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. find
  20. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 12–21. find
  21. Peng Shi and Yue Zhang. 2017. Joint bi-affine parsing and semantic role labeling. In Proceedings of the 2017 International Conference on Asian Language Processing (IALP), pages 338–341. find
  22. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038. find our card
  23. Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. 2019. Simplifying graph convolutional networks. arXiv:1902.07153. ↗
  24. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205–2215. find our card
  25. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45. find