Elamur Research⌕
☰
home / papers / 2020 / SciREX: A Challenge Dataset for Document-Level Information Extraction

SciREX: A Challenge Dataset for Document-Level Information Extraction

Summary

Most information extraction (IE) benchmarks work at the level of single sentences or short paragraphs, yet many facts in scientific papers can only be recovered by reading the whole article. The authors introduce SciREX, a document-level IE dataset of 438 full machine-learning papers. Each paper is annotated for entity mentions of four types (Dataset, Metric, Task, Method), coreference, entity saliency, and N-ary relations that describe the paper's main results. Annotating long texts by hand is expensive, so they combined several sources: result tuples from Papers with Code served as distant supervision, a SciERC-trained tagger proposed candidate mentions with high recall, and experts then corrected those candidates. They also built an end-to-end neural baseline. It uses a CRF tagger instead of span enumeration so that it scales to long inputs, and it then performs coreference, saliency filtering, and relation classification. The baseline outperforms existing models on the individual subtasks, but its end-to-end scores remain low. The analysis shows that detecting salient entities is the main bottleneck. SciREX therefore gives the field an open, demanding testbed for full-document IE.

Key points

  • Scale compared with SciERC: an average SciREX document contains about 5,737 words, 22 sections, 360 mentions, 8 salient entities, 16 binary relations and 5 4-ary relations. A SciERC abstract has about 130 words and 16 mentions.
  • Source data: the result tuples come from Papers with Code, which covers 1,170 ML articles annotated with (Dataset, Metric, Method, Task, Score). Raw text and section structure were extracted with LaTeXML.
  • Why the relations stay 4-ary: they cannot be decomposed into binary pairs, because a single dataset may serve several tasks, and each task can have its own metric.
  • Saliency is the bottleneck: supplying gold cluster saliency raises 4-ary relation performance by more than 10×. Correctly classified salient clusters average about 15 mentions, while misclassified ones average about 6, which suggests the model leans too heavily on mention frequency.
  • Availability: the data and code are released at github.com/allenai/SciREX.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.670

@inproceedings{jain2020scirex,
  title = {{SciREX: A Challenge Dataset for Document-Level Information Extraction}},
  author = {Jain, Sarthak and Zuylen, Madeleine van and Hajishirzi, Hannaneh and Beltagy, Iz},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.670},
  url = {https://doi.org/10.18653/v1/2020.acl-main.670}
}

References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Grobid. https://github.com/kermitt2/grobid . ↗
  2. Gabor Angeli, Victor Zhong, Danqi Chen, Arun Tejasvi Chaganty, Jason Bolton, Melvin Jose Johnson Premkumar, Panupong Pasupat, Sonal Gupta, and Christopher D. Manning. 2015. Bootstrapped self training for knowledge base population. Theory and Applications of Categories. find
  3. Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. find
  4. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). find our card
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015. find our card
  6. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. find our card
  7. Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find our card
  8. Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2017. Overview of linguistic resources for the tac kbp 2017 evaluations: Methodologies and results. Theory and Applications of Categories. find
  9. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. find our card
  10. Jeremy Getman, Joe Ellis, Stephanie Strassel, Zhiyi Song, and Jennifer Tracey. 2018. Laying the groundwork for knowledge base population: Nine years of linguistic resources for tac kbp. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  11. Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Flore… find
  12. Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level n-ary relation extraction with multi-scale representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor… find
  13. Zhanming Jie and Wei Lu. 2019. Dependency-guided LSTM-CRF for named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China. Association for Computational… find
  14. David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6. find our card
  15. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association… find our card
  16. Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi-supervised neural tagging. In Proceedings of Empirical Methods in Natural Language Processing. find
  17. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of The North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  18. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. Association for Computational Linguistics. find our card
  19. Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational… find
  20. Rashedur Rahman, Brigitte Grau, Sophie Rosset, Yoann Dupont, Jérémy Guillemot, Olivier Mesnard, Christian Lautier, and Wilson Fred. 2016. Tac kbp 2016 cold start slot filling and slot filler validation systems by irt systemx. Theory and Applications of Categories. find
  21. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer. find
  22. Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20. find
  23. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Li… find
  24. Chen-Tse Tsai, Gourab Kundu, and Dan Roth. 2013. Concept-based analysis of scientific literature. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management. find
  25. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… find our card
  26. Joe H. Ward. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301). find
  27. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jain-2020-scirex-challenge-dataset-document-level/