Elamur Research⌕
☰
home / papers / 2022 / Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction

Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction

Paper · 2022 · Qingyu Tan, Lu Xu, Lidong Bing et al.

Summary

DocRED is a standard benchmark for document-level relation extraction. It was built by having annotators filter candidate triples suggested by Wikidata distant supervision and an early RE model. The authors show that this process left many true relations unlabeled, so false negatives are widespread. They trace the gaps to three causes: Wikidata's sparse coverage, the limited quality of the original recommender model, and missing inverse relations such as symmetric sibling links. Because the development and test sets share these gaps, existing evaluations are unreliable.

To address this, they re-annotated 4,053 DocRED documents in an iterative human-in-the-loop process. Several current document-level RE models proposed missing triples, and human annotators verified them. The resulting dataset, Re-DocRED, contains more than twice as many relation triples as the original. State-of-the-art models trained and tested on Re-DocRED score roughly 13 F1 points higher. This suggests that much of the apparent performance ceiling on DocRED came from incomplete labels rather than model weakness.

Key points

  • Re-DocRED raises the triple count for the covered documents from 50,503 to 120,664, and the share of no_relation entity pairs drops from 97.1% to 94.0%. Coreference errors were also corrected.
  • The evaluation data went through two rounds of revision, while the training data received only one. The authors acknowledge that some relations may still be missing.
  • Compared with the contemporaneous from-scratch annotation by Huang et al. (2022), which covers 96 documents, Re-DocRED is much larger, more precise, has more triples per document, and scales more easily to new relation types.
  • Original DocRED has 5,053 Wikipedia documents, 96 relation types plus no_relation, and about 393.6 entity pairs per document. Re-annotation of sentence-level datasets (TACRED → Re-TACRED, Wiki20 → Wiki20m) shows a similar pattern of hidden false negatives.
  • The dataset is public at github.com/tonytan48/Re-DocRED. Annotators were paid about 60 CNY per hour.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations🏢Alibaba DAMO AcademyHangzhou · China28 works · 6,115 citations🏢Alibaba Group (Cayman Islands)George Town · Cayman Islands15 works · 3,209 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Information extraction · 109 works

Cite

Cite as Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng and Sharifah Mahani Aljunied. 2022. Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.580

@inproceedings{tan2022revisiting,
  title = {{Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction}},
  author = {Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng and Sharifah Mahani Aljunied},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.580},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.580}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of ACL. our card
  2. Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Proceedings of AAAI. find
  3. Lidong Bing, Sneha Chaudhari, Richard Wang, and William Cohen. 2015. Improving distant supervision for information extraction using label propagation through lists. In Proceedings of EMNLP. find
  4. Lidong Bing, Wai Lam, and Tak-Lam Wong. 2013. Wikipedia entity expansion and attribute extraction from the web using semi-supervised learning. In Proceedings of WSDM. find
  5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of ACM SIGMOD. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NIPS. our card
  7. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of LREC. our card
  8. Jhih-wei Chen, Tsu-Jui Fu, Chen-Kang Lee, and Wei-Yun Ma. 2021. H-FND: hierarchical false-negative denoising for distant supervision relation extraction. In Findings of ACL. find
  9. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin. find
  10. Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction. In Findings of ACL. find
  11. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of EMNLP. find
  12. Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun. 2020. More data, more relations, more context and more openness: A review and outlook for relation extraction. In Proceedings of AACL. find
  13. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP. find
  14. Kailong Hao, Botao Yu, and Wei Hu. 2021. Knowing false negatives: An adversarial training method for distantly supervised relation extraction. In Proceedings of EMNLP. find
  15. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Wo… our card
  16. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of ACL. find
  17. Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-revise produce reliable annotations? an analysis on missing instances in DocRED. In Proceedings of ACL. find
  18. Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022. RelationPrompt: leveraging prompts to generate synthetic data for zero-shot relation triplet extraction. Findings of ACL. our card
  19. Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems. our card
  20. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. find
  21. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia – a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web. our card
  22. Yangming Li, Lemao Liu, and Shuming Shi. 2021. Empirical analysis of unlabeled entity problem in named entity recognition. In Proceedings of ICLR. find
  23. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021. MulDA: A multilingual data augmentation framework for low-resource cross-lingual NER. In Proceedings of ACL. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  25. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL. find
  26. Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of ACL. our card
  27. Tapas Nayak and Hwee Tou Ng. 2019. Effective attention modeling for neural relation extraction. In Proceedings of CoNLL. find
  28. Tapas Nayak and Hwee Tou Ng. 2020. Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In Proceedings of AAAI. find
  29. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of NAACL. find
  30. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178. our card
  31. Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document-level distant supervision. In Proceedings EMNLP. find
  32. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. In Findings of EMNLP. our card
  33. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: a large-scale document-level relation extraction dataset. In Proceedings of ACL. find
  34. Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. Proceedings of IJCAI. find
  35. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP. find
  36. Evan Sandhaus. 2008. The New York Times annotated corpus. Linguistic Data Consortium. find
  37. Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. MELM: Data augmentation with masked entity language modeling for low-resource NER. In Proceedings of ACL. find
  38. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Proceedings of EMNLP. our card
  39. Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction. arXiv preprint arXiv:2102.01373. our card
  40. George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: addressing shortcomings of the tacred dataset. In Proceedings of AAAI. find
  41. Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of AAAI. find
  42. Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of ACL. our card
  43. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM. find
  44. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  45. ACE 2005 multilingual training corpus. Linguistic Data Consortium. find
  46. Zihao Wang, Kwunping Lai, Piji Li, Lidong Bing, and Wai Lam. 2019. Tackling long-tailed relations and uncommon entities in knowledge graph completion. In Proceedings of EMNLP. find
  47. Example 1 Error Type 1: Misunderstanding of Definition: creator vs. architect Error Cause: Annotator’s Misunderstanding of Wikidata Relation Definition Error Analysis: There are over 9,000 relation types in the Wikidata knowledge base, whereas DocRED only contains 84 of them. The relation between bu… find
  48. Wrong Triples: (Museo Jumex, creator, David Alan Chipperfield), (River and Rowing Museum, creator, David Alan Chipperfield), (Museum of Modern Literature, creator, David Alan Chipperfield), (Saint Louis Art Museum,creator, David Alan Chipperfield), (Hepworth Wakefield gallery, creator, David Alan Ch… find
  49. Wrong Triples: (Taking Over Me, series, Chapman Square Chapter II), (Brokenhearted, series, Chapman Square Chapter II), (When She Was Mine, series, Chapman Square), (Standing in the Dark, series, Chapman Square) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/