Elamur Research⌕
☰
home / papers / 2020 / Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Paper · 2020 · Hao Peng, Tianyu Gao, Xu Han et al.

Summary

The paper asks what neural relation extraction (RE) models actually use when they classify the relation between two entities: the surrounding sentence or the entity names. The authors ran controlled input-ablation experiments on TACRED with CNN, BERT and a re-implemented MTB. Inputs were full sentences, sentences with entities replaced by their types, context only, mentions only, or types only. Context turns out to be the main signal, but mentions also matter a lot, and mostly because they reveal entity type. Models also score surprisingly well from mentions alone, which suggests that benchmarks leak shallow, entity-based shortcuts.

Based on this, the authors propose CP, an entity-masked contrastive pre-training framework. It uses Wikidata to collect sentences that likely express the same relation. A contrastive objective pulls these relationally similar sentences together, and entity mentions are randomly masked so the model cannot memorise names. CP beats BERT and MTB on several benchmarks, including biomedical ones, with the largest gains in low-resource and few-shot settings. The work points to better pre-training as a route to more robust RE.

Key points

  • Pilot results (TACRED, micro F1, BERT):

    • Full sentence (C+M): 0.683
    • Context+Type (C+T): 0.686
    • Context only: 0.570
    • Mentions only: 0.466
    • Types only: 0.277

    C+T roughly matching C+M shows that most of the useful mention information is type.

  • Dataset: TACRED, the pilot dataset, has 106,264 instances and 42 relations, with entity type annotations.

  • CNN behaviour: CNN overfits mention cues more readily than BERT or MTB. Its context-only and mention-only scores are nearly equal (0.441 vs 0.434).

  • Evaluation: CP was tested on TACRED, Wiki80, ChemProt, FewRel 1.0 and FewRel 2.0 (domain adaptation to biomedical text). It improved the full-sentence, context-only and mention-only settings.

  • Low-resource gains in the full-sentence setting:

    • about +7% on FewRel 1.0 10-way 1-shot
    • +18% on TACRED with 1% of the training data
    • +24% on Wiki80 with 1% of the training data

    The code and data are available at github.com/thunlp/RE-Context-or-Names.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.298

@inproceedings{peng2020learning,
  title = {{Learning from Context or Names? An Empirical Study on Neural Relation Extraction}},
  author = {Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun and Jie Zhou},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.298},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.298}
}

References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of ACL.
  2. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of ACL, pages 2895–2905.
  3. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In Proceedings of EMNLP, pages 615–620. find
  4. Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of EMNLP, pages 724–731.
  5. Mary Elaine Califf and Raymond J. Mooney. 1997. Relational learning of pattern-match rules for information extraction. In Proceedings of CoNLL. find
  6. Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In Proceedings of ACL, page 423.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  8. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of EMNLP-IJCNLP, pages 6251–6256. find
  9. Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun. 2019. OpenNRE: An open and extensible toolkit for neural relation extraction. In Proceedings of EMNLP-IJCNLP: System Demonstrations, pages 169–174. find
  10. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP, pages 4803–4809. find
  11. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman…
  12. Scott B Huffman. 1995. Learning information extraction patterns from examples. In Proceedings of IJ-CAI, pages 246–260. find
  13. Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In Proceedings of ACL, pages 178–181. find
  14. Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. 2016. ChemProt-3.0: A global chemical biology diseases mapping. find
  15. Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of ACL, pages 2124–2133. find
  16. Chunyang Liu, Wenbo Sun, Wenhan Chao, and Wanxiang Che. 2013. Convolution neural network for relation extraction. In Proceedings of ICDM, pages 231–242. find
  17. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: Enabling language representation with knowledge graph. In Proceedings of AAAI, pages 2901–2908.
  18. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR 2019. find
  19. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE. find
  20. Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of ACL, pages 1468–1478.
  21. Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. More data, more relations, more context and more openness: A review and outlook for relation extraction. find
  22. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. find
  23. Bonan Min, Ralph Grishman, Li Wan, Chang Wang, and David Gondek. 2013. Distant supervision for relation extraction with an incomplete knowledge base. In Proceedings of NAACL, pages 777–782.
  24. Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of EMNLP, pages 1753–1762. find
  25. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL-IJCNLP, pages 1003–1011. find
  26. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 2335–2344.
  27. Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the NAACL Workshop on Vector Space Modeling for NLP, pages 39– 48.
  28. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. arXiv preprint arXiv:1508.01006. find
  29. Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of EMNLP-IJCNLP, pages 43–54. find
  30. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of EMNLP-IJCNLP, pages 2463–2473.
  31. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP, pages 35–45. find
  32. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of ACL, pages 1441–1451. find
  33. Guodong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of ACL, pages 427–434.
  34. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of ECML-PKDD, pages 148–163. find
  35. Dan Roth and Wen-tau Yih. 2002. Probabilistic reasoning for entity & relation recognition. In Proceedings of COLING. find
  36. Dan Roth and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Proceedings of CoNLL.
  37. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Proceedings of NIPS, pages 4077–4087. find
  38. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP, pages 1201–1211.
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998– 6008.
  40. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Proceedings of CACM, 57(10):78–85. find
  41. Chenyan Xiong, Russell Power, and Jamie Callan. 2017. Explicit semantic ranking for academic search via knowledge graph embedding. In Proceedings of WWW, pages 1271–1279. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2020-learning-context-or-names-empirical/