Elamur Research⌕
☰
home / papers / 2017 / SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications

SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications

Summary

ScienceIE, SemEval 2017 Task 10, is a shared task on finding and typing keyphrases in scientific text and linking them with semantic relations. The goal is to show which papers deal with which tasks, processes and materials. The organisers built a corpus of 500 paragraphs from open-access ScienceDirect articles, split evenly across Computer Science, Material Sciences and Physics. Each paragraph was double-annotated in BRAT, with an expert resolving disagreements. The task has three linked subtasks: (A) marking keyphrase boundaries, (B) labelling each mention as PROCESS, TASK or MATERIAL, and (C) extracting HYPONYM-OF and SYNONYM-OF relations between mentions. Systems were scored by exact match with micro-averaged precision, recall and F1. Three evaluation scenarios gave teams different amounts of gold input. The task drew 26 submissions. Boundary detection (Subtask A) proved hardest, because many keyphrases are long and rare. The resource matters because its hypernym and synonym links are annotated at mention level, whereas existing resources such as WordNet work only at type level.

Key points

  • Data split: 350 training, 50 development and 100 test paragraphs.
  • Training-set statistics:
    • 5730 keyphrase mentions, of which 1697 are unique.
    • 31% of keyphrases occur only once.
    • 22% are 5 or more tokens long.
    • 93% are noun phrases, according to the nltk tagger.
  • Annotation process and agreement:
    • Student annotators were paid per document (about 10 GBP/hour, roughly 12 minutes per document).
    • Payment was made only after all assigned documents were finished.
    • Cohen's kappa against the expert ranged from 0.45 to 0.85 among students who completed the work.
    • Several annotators whose quality declined over time scored far lower.
  • Scenario 2 (gold Subtask A provided):
    • MayoNLP ranked first with an SVM using rich lexical and orthographic features (0.64 overall F1).
    • UKP/EELECTION, a majority-vote ensemble of three neural models, came close at 0.63.
    • The upper bound was 0.84 and the random baseline 0.15.
  • Scenario 3 (only relations predicted):
    • MIT ranked first with a CNN plus rule-based post-processing and argument ordering (0.64 F1).
    • Most teams in this scenario beat all relation results from the scenarios without gold keyphrases and types.

Interesting ideas

  • I1. Staged evaluation scenarios: The same pipeline was evaluated end-to-end, with gold boundaries supplied, and with gold boundaries and types supplied. This isolates how errors from earlier stages propagate. The design suits any multi-step IE benchmark, such as NER followed by relation extraction on Ukrainian text.
  • I2. Cheap paragraph selection: Instead of annotating whole papers, the organisers used a coarse embedding-based method to find keyphrase-dense paragraphs, then picked by hand those likely to contain relations. This concentrates annotation budget where rare relations actually occur.
  • I3. Character-offset stand-off annotation: Annotations are stored separately from the raw text, as spans defined by character offsets. This keeps the text untouched and makes exact-match scoring straightforward. It also handles long multi-token spans that BIO tagging makes awkward.
  • I4. Quality control for paid annotators: The organisers paired student annotators with an expert who adjudicated disagreements. They also excluded unfinished documents when computing kappa and conditioned payment on completing all assigned documents. They noted that quality can degrade over time and recommend stronger screening before annotation starts, which is a useful lesson for any crowdsourced annotation project.
  • I5. Simple statistics as cheap heuristics: Reporting that 93% of keyphrases are noun phrases immediately suggests a candidate-generation step based on noun-phrase chunking. The high share of singleton keyphrases warns that systems relying on memorised lists will fail. Profiling a dataset this way helps choose a modelling approach early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

118 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2091

@inproceedings{augenstein2017semeval,
  title = {{SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications}},
  author = {Augenstein, Isabelle and Das, Mrinal and Riedel, Sebastian and Vikraman, Lakshmi and McCallum, Andrew},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2091},
  url = {https://doi.org/10.18653/v1/s17-2091}
}

References

45 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kazi Saidul Hasan and Vincent Ng. 2014. Automatic Keyphrase Extraction: A Survey of the State of the Art. In Proceedings of ACL. find our card
  2. Yuridiana Alemán, Darnes Vilariño, and Josefa So-modevilla. 2017. BUAP at SemEval-2017 Task 10. In Proceedings of SemEval. find
  3. Marti Hearst. 1991. Noun Homograph Disambiguation Using Local Context in Large Text Corpora. Using Corpora pages 185–188. find
  4. Waleed Ammar, Matthew Peters, Chandra Bhagavatula, and Russell Power. 2017. The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction. In Proceedings of SemEval. find
  5. Simon David Hernandez, Davide Buscaldi, and Thierry Charnois. 2017. LIPN at SemEval-2017 Task 10: Selecting Candidate Phrases with Part-of-Speech Tag Sequences from the training data. In Proceedings of SemEval. find
  6. Isabelle Augenstein and Anders Søgaard. 2017. Multi-Task Learning of Keyphrase Boundary Classification. In Proceedings of ACL. find
  7. Dain Kaplan, Ryu Iida, and Takenobu Tokunaga. 2009. Automatic Extraction of Citation Contexts for Research Paper Summarization: A Coreference-chain based Approach. In Proceedings of NLPIR4DL workshop at ACL-IJCNLP. find
  8. Isabelle Augenstein, Andreas Vlachos, and Diana Maynard. 2015. Extracting Relations between Non-Standard Entities using Distant Supervision and Imitation Learning. In Proceedings of EMNLP. find
  9. Rohit J Kate and Raymond J Mooney. 2010. Joint Entity and Relation Extraction using Card-Pyramid Parsing. In Proceedings of CoNLL. find
  10. Biswanath Barik and Erwin Marsi. 2017. NTNU-2@ScienceIE: Identifying Synonym and Hyponym Relations among Keyphrases in Scientific Documents. In Proceedings of SemEval. find
  11. Roman Kern, Stefan Falk, and Andi Rexha. 2017. Know-Center at SemEval-2017 Task 10: CODE annotator. In Proceedings of SemEval. find
  12. Gábor Berend. 2017. SZTE-NLP at SemEval-2017 Task 10: A High Precision Sequence Model for Keyphrase Extraction Utilizing Sparse Coding for Feature Generation. In Proceedings of SemEval. find
  13. Su Nam Kim, Timothy Baldwin, and Min-Yen Kan. 2010a. Evaluating n-gram based evaluation metrics for automatic keyphrase extraction. In Proceedings of Coling. find
  14. Panot Chaimongkol, Akiko Aizawa, and Yuka Tateisi. 2014. Corpus for Coreference Resolution on Scientific Papers. In Proceedings of LREC. find
  15. Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010b. SemEval-2010 Task 5 : Automatic Keyphrase Extraction from Scientific Articles. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Uppsala, Sweden, pages 21– 26. find our card
  16. Steffen Eger, Erik-Lân Do Dinh, Ilia Kuznetsov, Masoud Kiaeeha, and Iryna Gurevych. 2017. EELEC-TION at SemEval-2017 Task 10: Ensemble of nEural Learners for kEyphrase ClassificaTION. In Proceedings of SemEval. find
  17. Ji Young Lee, Franck Dernoncourt, and Peter Szolovits. 2017a. MIT at SemEval-2017 Task 10: Relation Extraction with Convolutional Neural Networks. In Proceedings of SemEval. find
  18. Gerardo Flores, Mireya Tovar, and José A. Reyes-Ortiz. 2017. GMBUAP at SemEval-2017 Task 10: A First Approach Based Empirical Rules and n-grams for the Extraction and Classification of Keyphrases. In Proceedings of SemEval. find
  19. Lung-Hao Lee, Kuei-Ching Lee, and Yuen-Hsien Tseng. 2017b. The NTNU System at SemEval-2017 Task 10: Extracting Keyphrases and Relations from Scientific Publications Using Multiple Conditional Random Fields. In Proceedings of SemEval. find
  20. Sujatha Das Gollapalli and Xiaoli Li. 2015. EMNLP versus ACL: Analyzing NLP research over time. In Proceedings of EMNLP. find
  21. Kartik Goyal and Chris Dyer. 2016. Posterior regularization for Joint Modelling of Multiple Structured Prediction Tasks with Soft Constraints. In Proceedings of EMNLP. find
  22. Sijia Liu, Shen Feichen, Vipin Chaudhary, and Hongfang Liu. 2017. MayoNLP at SemEval 2017 Task 10: Word Embedding Distance Pattern for Keyphrase Classification in Scientific Publications. In Proceedings of SemEval. find
  23. Yufan Guo, Roi Reichart, and Anna Korhonen. 2015. Unsupervised Declarative Knowledge Induction for Constraint-Based Learning of Information Structure in Scientific Documents. Transactions of the Association for Computational Linguistics 3:131–143. find
  24. Erwin Marsi and Pinar Öztürk. 2015. Extraction and generalisation of variables from scientific publications. In Proceedings of EMNLP. find
  25. Erwin Marsi, Utpal Kumar Sikdar, Cristina Marco, Biswanath Barik, and Rune Sætre. 2017. NTNU-1@ScienceIE at SemEval-2017 Task 10: Identifying and Labelling Keyphrases with Conditional Random Fields. In Proceedings of SemEval. find
  26. Sonal Gupta and Christopher Manning. 2011. Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers. In Proceedings of IJC-NLP. find
  27. Vı́ctor Suárez-Paniagua, Isabel Segura-Bedmar, and Paloma Martı́nez. 2017. LaBDA Team at SemEval-2017 Task C: Relation Classification between keyphrases via Convolutional Neural Network. In Proceedings of SemEval. find
  28. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS. find
  29. M. Mintz, S. Bills, R. Snow, and D. Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL-IJCNLP. find
  30. Yuka Tateisi, Yo Shidahara, Yusuke Miyao, and Akiko Aizawa. 2014. Annotation of Computer Science Papers for Semantic Relation Extraction. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncin Moreno, Jan Odijk, and Stelios Piperidis, editor… find
  31. Diarmuid Ó Séaghdha and Simone Teufel. 2014. Unsupervised learning of rhetorical structure with un-topic models. In Proceedings of Coling. find
  32. Fuchun Peng and Andrew McCallum. 2006. Information extraction from research papers using conditional random fields. Information Processing and Management 42(4):963–979. find
  33. Simone Teufel. 2006. Argumentative Zoning for Improved Citation Indexing. In James G. Shanahan, Yan Qu, and Janyce Wiebe, editors, Computing Attitude and Affect in Text, Springer, volume 20 of The Information Retrieval Series, pages 159–169. find
  34. Animesh Prasad and Min-Yen Kan. 2017. WING-NUS at SemEval-2017 Task 10: Keyphrase Extraction and Classification as Joint Sequence Labeling. In Proceedings of SemEval. find
  35. Tomoki Tsujimura, Makoto Miwa, and Yutaka Sasaki. 2017. TTI-COIN at SemEval-2017 Task 10: Investigating Embeddings for End-to-End Relation Extraction from Scientific Papers. In Proceedings of SemEval. find
  36. Behrang QasemiZadeh and Anne-Kathrin Schumann. 2016. The ACL RD-TEC 2.0: A Language Resource for Evaluating Term Extraction and Entity Recognition Methods. In Proceedings of LREC. find
  37. Liang Wang and Sujian Li. 2017. PKU ICL at SemEval-2017 Task 10: Keyphrase Extraction with Model Ensemble and External Knowledge. In Proceedings of SemEval. find
  38. Dragomir Radev and Amjad Abu-Jbara. 2012. Rediscovering ACL Discoveries Through the Lens of ACL Anthology Network Citing Sentences. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries. find
  39. Torsten Zesch and Iryna Gurevych. 2009. Approximate Matching for Evaluating Keyphrase Extraction. In Proceedings of RANLP. find
  40. Isabel Segura-Bedmar, Cristóbal Colón-Ruiz, and Paloma Martı́nez. 2017. LABDA at SemEval-2017 Task 10: Extracting Keyphrases from Scientific Publications by combining the BANNER tool and the UMLS Semantic Network. In Proceedings of SemEval. find
  41. Yanchuan Sim, Bryan Routledge, and Noah A. Smith. 2015. A Utility Model of Authors in the Scientific Community. In Proceedings of EMNLP. find
  42. Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. 2013. Joint Inference of Entities, Relations, and Coreference. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction (AKBC). find
  43. R. Snow, D. Jurafsky, and A.Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. Advances in Neural Information Processing Systems 17:1297–1304. find
  44. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: a Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of EACL System Demonstrations. find our card
  45. Lucas Sterckx, Cornelia Caragea, Thomas Demeester, and Chris Develder. 2016. Supervised Keyphrase Extraction as Positive Unlabeled Learning. In Proceedings of EMNLP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/augenstein-2017-semeval-2017-task-10-scienceie/