Elamur Research⌕
☰
home / papers / 2020 / Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders

Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders

Summary

Joint extraction of named entities and relations is often framed as table filling. An N×N grid is built over word pairs: diagonal cells hold BIO entity tags, and off-diagonal cells hold directed relation labels. Earlier joint systems usually relied on one shared representation for both subtasks, which can blur the features each task needs. Many also flattened the table into a sequence and lost its 2D structure.

Wang and Lu instead build two cooperating encoders: a sequence encoder for NER and a table encoder for RE. In each stacked layer, the table is formed from pairs of sequence vectors. A multi-dimensional GRU-based RNN then contextualizes it across layer, row and column. The sequence encoder updates itself through table-guided attention. The authors also inject BERT-family self-attention weights into the table as signals of word-word interaction, which they describe as a first.

The model achieves state-of-the-art F1 for both tasks on ACE04, ACE05, CoNLL04 and ADE. This shows that separate but interacting task-specific representations are beneficial.

Key points

  • Architecture: Multiple layers alternate between building the table from the sequence representation and refining the sequence via table-guided attention. Input embeddings combine word vectors, character-level LSTM vectors, and contextualized vectors from models such as BERT or ALBERT.
  • MD-RNN ablation on ACE05: Results are reported as two scores per setting, presumably NER and RE F1.
    • The default bidirectional MD-RNN scores 89.5 / 67.6.
    • The four-direction variant gives 89.7 / 67.6.
    • The unidirectional variant drops RE to 66.9.
    • Keeping only the layer dimension yields 63.9 on RE.
    • CNN-based table encoders performed worse than RNN-based ones.
  • Attention visualization: Using bertviz, the authors compare table-guided attention heads with ALBERT heads. The table-guided heads capture entity and relation patterns that ALBERT misses, and do so with higher confidence.
  • Probing intermediate layers: The trained prediction layer is applied to intermediate layers on ACE05 examples. Early-layer errors are progressively corrected in deeper layers.
  • Code: Released at github.com/LorrinWWW/two-are-better-than-one.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

70 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.133

@inproceedings{wang2020two,
  title = {{Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders}},
  author = {Wang, Jue and Lu, Wei},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.133},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.133}
}

References

62 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proc. of NAACL-HLT. find our card
  2. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint. find our card
  3. George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie M. Strassel, and Ralph M. Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Proc. of LREC. find our card
  4. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proc. of ICLR. find our card
  5. Markus Eberts and Adrian Ulges. 2019. Span-based joint entity and relation extraction with transformer pre-training. arXiv preprint. find
  6. Radu Florian, Hongyan Jing, Nanda Kambhatla, and Imed Zitouni. 2006. Factorizing complex models: A case study in mention detection. In Proc. of ACL. find
  7. Radu Florian, John F. Pitrelli, Salim Roukos, and Imed Zitouni. 2010. Improving mention detection robustness to noisy input. In Proc. of EMNLP. find
  8. Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2007. Multi-dimensional recurrent neural networks. In Proc. of ICANN. find
  9. Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proc. of ICASSP. find
  10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  11. Pankaj Gupta, Subburam Rajaram, Hinrich Schütze, and Thomas Runkler. 2019. Neural relation extraction within and across sentence boundaries. In Proc. of AAAI. find
  12. Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018a. Adversarial training for multi-context joint entity and relation extraction. In Proc. of EMNLP. find
  13. Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Proc. of COLING. find
  14. Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018b. Joint entity recognition and relation extraction as a multi-head selection problem. Expert Systems with Applications. find
  15. Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In Proc. of NAACL-HLT. find
  16. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proc. of EMNLP. find
  17. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT. find
  19. Kalpit Dixit and Yaser Al. 2019. Span-level model for relation extraction. In Proc. of ACL. find
  20. Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of biomedical informatics. find
  21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proc. of CVPR. find
  22. Jing Jiang and ChengXiang Zhai. 2007. A systematic exploration of the feature space for relation extraction. In Proc. of HLT-NAACL. find
  23. Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Proc. of ACL. find
  24. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proc. of HLT-NAACL. find
  25. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In Proc. of ICLR. find
  26. Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. 2017. A neural joint model for entity and relation extraction from biomedical text. BMC bioinformatics. find
  27. Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji. 2016. Joint models for extracting adverse drug events from biomedical text. In Proc. of IJCAI. find
  28. Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proc. of ACL. find our card
  29. Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Proc. of ACL. find our card
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint. find
  31. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proc. of NAACL-HLT. find our card
  32. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In Proc. of ACL. find our card
  33. Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In Proc. of EMNLP. find our card
  34. Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In Proc. of ACL. find our card
  35. Dat Quoc Nguyen and Karin Verspoor. 2019. End-to-end neural relation extraction using deep biaffine attention. In Proc. of ECIR. find
  36. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics. find our card
  37. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proc. of EMNLP. find
  38. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. find
  39. Barbara Plank and Alessandro Moschitti. 2013. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In Proc. of ACL. find
  40. Lev-Arie Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proc. of CoNLL. find our card
  41. Dan Roth and Wen tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Proc. of CoNLL. find our card
  42. Erik F. Sang and Jorn Veenstra. 1999. Representing text chunks. In Proc. of EACL. find our card
  43. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proc. of EMNLP. find our card
  44. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint. find our card
  45. Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011. Semi-supervised relation extraction with large-scale word clustering. In Proc. of NAACL-HLT. find
  46. Changzhi Sun, Yuanbin Wu, Man Lan, Shiliang Sun, Wenting Wang, Kuang-Chih Lee, and Kewen Wu. 2018. Extracting entities and relations with joint minimum risk training. In Proc. of EMNLP. find
  47. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. of ACL. find
  48. Tung Tran and Ramakanth Kavuluru. 2019. Neural metric learning for fast end-to-end relation extraction. arXiv preprint. find
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS. find
  50. Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In Proc. of NAACL-HLT. find
  51. Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proc. of ACL. find our card
  52. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proc. of EMNLP/IJCNLP. find our card
  53. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  54. Ace 2005 multilingual training corpus. Linguistic Data Consortium. find
  55. Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar. 2019. Extracting multiple-relations in one-pass with pre-trained transformers. In Proc. of ACL. find
  56. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. find
  57. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint. find our card
  58. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In Proc. of ACL. find
  59. Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation extraction. In Proc. of EMNLP. *Figure 8: For 2D-RNNs, cells in the same color can be computed in parallel.* And then calculate the hidden states: T̃ l,i,j = tanh(X l,i,j W x + r l,i,j Daojian Zeng, Kang Liu, S… find our card
  60. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. arXiv preprint. find
  61. Meishan Zhang, Yue Zhang, and Guohong Fu. 2017. End-to-end neural relation extraction with global optimization. In Proc. of EMNLP. find
  62. Shubin Zhao and Ralph Grishman. 2005. Extracting relations with integrated information using kernel methods. In Proc. of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-two-are-better-than-one/