Elamur Research⌕
☰
home / papers / 2018 / SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers

SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers

Paper · 2018 · Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann et al.

Summary

The paper reports on a SemEval-2018 shared task on recognising semantic relations between domain concepts in computational linguistics abstracts. The goal is to support literature search that keyword engines cannot handle, such as finding which method was applied to which task or data. The organisers built a dataset from the ACL Anthology Reference Corpus. They drew on two sources: ACL RD-TEC 2.0, with manually annotated entities, and ACL-RelAcS, with automatically annotated entities. Relations were labelled with six coarse types derived from a finer annotation scheme. The task had three subtasks: classification with manual entities, classification with automatic (noisy) entities, and joint extraction plus classification. This design lets the effect of preprocessing quality be measured directly. The task drew 32 participants and 158 submissions, mostly CNN and LSTM systems with word embeddings, often trained on in-domain text. Classifying given relation pairs worked well, but finding the pairs remained hard. Surprisingly, systems scored higher on the noisy entities than on the clean ones, which shows that annotation choices strongly shape the measured results.

Key points

  • The six relation types are USAGE, RESULT, MODEL-FEATURE, PART_WHOLE, TOPIC and COMPARE. All are directional, with ARG1 and ARG2 roles.
  • Each subtask had 350 training abstracts and 150 test abstracts, averaging about 100 words each. Subtasks 1.1 and 2 shared the same training data, and each team could submit three runs per subtask.
  • The official metric for classification was macro-averaged F1. Subtask 2 was scored in two stages: first pair extraction, ignoring labels and direction, then classification of the correctly found pairs.
  • Average F1 of the top 5 systems by relation:
    • In subtask 1.1, USAGE was easiest (0.85) and COMPARE hardest (0.59).
    • In subtask 1.2, TOPIC reached 0.97, while MODEL-FEATURE was lowest (0.79).
  • Human entity annotation reached acceptable agreement (κ > 0.7) only after training and guideline refinement. UWNLP, the only system treating full specialised terms as semantic units, won the extraction subtask.

Interesting ideas

  • I1. Controlled preprocessing comparison. The task offers the same problem with gold entities, automatic entities, and no given pairs. This isolates how much each pipeline stage costs. A similar layered setup would help benchmark any IE pipeline, for example on Ukrainian texts, where tokenisation or NER quality can dominate the results.
  • I2. "Noisy" data can be easier. Automatic entities were shorter and less specific, and they sometimes included verbal cues signalling the relation. This made classification easier but arguably less useful for people. It is a reminder to check whether a higher score reflects real ability or simply friendlier annotation.
  • I3. Per-class type-token ratio as a difficulty signal. Counting distinct entity pairs per relation class showed that low-variety classes such as TOPIC are easy to learn. This cheap diagnostic can explain uneven per-class scores in any classification dataset.
  • I4. Pre-annotation followed by boundary correction. Annotators corrected automatic entity spans instead of starting from scratch, with a training phase and individual feedback beforehand. This speeds up expert annotation in specialised domains while keeping agreement acceptable.
  • I5. Fine-grained scheme collapsed into coarse labels. Annotating with detailed subtypes and then merging them into six classes keeps the richer information for later use while giving systems a tractable label set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France104 works · 60,877 citations🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany14 works · 1,132 citations🔬Laboratoire d'Informatique de Paris-NordVilletaneuse · France3 works · 405 citations🎓Université Sorbonne Paris NordVilletaneuse · France4 works · 377 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksBiochemistry, Genetics and Molecular Biology · 116 worksMolecular Biology · 114 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Haïfa Zargayouna and Thierry Charnois. 2018. SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1111

@inproceedings{gabor2018semeval,
  title = {{SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers}},
  author = {Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Haïfa Zargayouna and Thierry Charnois},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1111},
  url = {https://doi.org/10.18653/v1/s18-1111}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. find
  2. Semeval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546–555, Vancouver, Canada. Association for Computational Linguistics.
  3. Isabelle Augenstein and Anders Søgaard. 2017. Multi-task learning of keyphrase boundary classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). find
  4. Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics. In Proceedings of the 6th International Co…
  5. Georgeta Bordea. 2013. Domain Adaptive Extraction of Topical Hierarchies for Expertise Mining. Phd thesis, National University of Ireland, Galway. find
  6. Georgeta Bordea, Paul Buitelaar, and Tamara Polajnar. 2013. Domain-independent term extraction through domain modelling. In 10th International Conference on Terminology and Artificial Intelligence (TIA 2013). find
  7. Walter Daelemans and Antal van den Bosch. 2005. Memory-Based Language Processing. Studies in Natural Language Processing. Cambridge University Press. find
  8. Beatrice Daille, Christine Jacquin, Laura Monceaux, Emmanuel Morin, and Jerome Rocheteau. 2013. TTC TermSuite : Une chaine de traitement pour la fouille terminologique multilingue. In Proceedings of the Traitement Automatique des Langues Naturelles Conference (TALN). find
  9. Kata Gábor, Hafa Zargayouna, Davide Buscaldi, Isabelle Tellier, and Thierry Charnois. 2016a. Semantic annotation of the ACL Anthology Corpus for the automatic analysis of scientific literature. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), pages 3694–37… find
  10. Kata Gábor, Hafa Zargayouna, Isabelle Tellier, Davide Buscaldi, and Thierry Charnois. 2016b. A typology of semantic relations dedicated to scientific literature analysis. In SAVE-SD Workshop at the 25th World Wide Web Conference. Lecture Notes in Computer Science 9792, pages 26–32. find
  11. Marti Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING ’92, pages 539–545. find
  12. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Saghdha, Sebastian Pad, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Eva…
  13. David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. SemEval-2012 Task 2: Measuring degrees of relational similarity. In Proceedings of the Workshop on Semantic Evaluations.
  14. Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles. In Proceedings of SemEval 2010.
  15. Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi-supervised neural tagging. In EMNLP 2017. find
  16. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  17. F. Osborne and E. Motta. 2015. Klink-2: Integrating multiple web sources to generate semantic topic networks. In Proceedings of the 14th International Conference on The Semantic Web - ISWC 2015 - Volume 9366, pages 408–424, New York, NY, USA. Springer-Verlag New York, Inc. find
  18. Behrang QasemiZadeh and Laura Kallmeyer. 2016. Random Positive-Only Projections: PPMI-enabled incremental semantic space construction. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 189–198, Berlin, Germany. Association for Computational Linguistics. find
  19. Behrang QasemiZadeh and Anne-Kathrin Schumann. 2016. The ACL RD-TEC 2.0: A language resource for evaluating term extraction and entity recognition methods. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find
  20. Anne-Kathrin Schumann and Behrang QasemiZadeh. 2015. The ACL RD-TEC Annotation Guideline: A Reference Dataset for the Evaluation of Automatic Term Recognition and Classification. Technical report. find
  21. Chen-Tse Tsai, Gourab Kundu, and Dan Roth. 2013. Concept-based analysis of scientific literature. In ACM Conference on Information and Knowledge Management ACM, pages 1733–1738. find
  22. Dietmar Wolfram. 2016. Bibliometrics, information retrieval and natural language processing: Natural synergies to support digital library research. In Proceedings of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL), page… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gabor-2018-semeval-2018-task-7-semantic/