Elamur Research⌕
☰
home / papers / 2013 / SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013)

SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013)

Paper · 2013 · Isabel Segura-Bedmar, Paloma Martínez and María Herrero-Zazo

Summary

DDIExtraction 2013 (SemEval-2013 Task 9) is a shared task on mining drug-drug interactions (DDIs) from biomedical literature, which matters for patient safety. Unlike the 2011 edition, where drug mentions were given to participants, this edition covers the whole pipeline in two subtasks. Task 9.1 recognizes and types pharmacological substances. Task 9.2 detects interacting drug pairs and labels the interaction type. Both rely on the DDI corpus, a manually annotated collection that mixes DrugBank texts with MedLine abstracts, so systems face two writing styles. The organizers defined a span-offset output format and a multi-criteria evaluation scheme for entity recognition. They also reported both micro- and macro-averaged scores.

Fourteen teams submitted 38 runs. The best F1 was 71.5% for Task 9.1 and 65.1% for interaction classification in Task 9.2. Simply detecting interactions reached 80% F1, up from roughly 66% in 2011. Results on MedLine were clearly lower than on DrugBank. The task provides a reusable benchmark, and the test gold labels are held back to keep later comparisons fair.

Key points

  • DDI corpus: 1,017 documents (784 DrugBank, 233 MedLine) with 18,491 annotated pharmacological substances and 5,021 DDIs. It is distributed as XML in the unified PPI corpus format of Pyysalo et al. About 77% of documents were used for training.
  • Entity types: drug, brand, group and drug-n (substances not approved for human use).
  • Interaction types: mechanism, effect, advice and int. The int class is rare, at about 5.6% of relations.
  • Task 9.1 results: 6 teams submitted 16 runs. The best system, from the WBI team, was a CRF using ChemSpot-derived features and domain resources. Second place went to a dictionary approach built on DrugBank, ATC and MeSH.
  • Difficulty by class:
    • Brand names were the easiest entities, while drug-n was nearly unrecognized.
    • Advice was the easiest interaction class (best F1 69.2%) and int the hardest (54.7%).
    • Non-linear kernel methods outperformed linear SVMs for interaction extraction.

Interesting ideas

  • I1. Scoring span and type separately: Entity recognition was scored under four criteria: strict, exact boundary only, partial boundary, and type with overlap. This separates boundary errors from typing errors, which helps when a downstream task can still use imperfect spans.
  • I2. Micro- plus macro-averaging: Reporting both shows how dominant classes inflate overall scores and exposes weak rare classes such as drug-n or int. This is useful for any imbalanced label set.
  • I3. Two text sources in one benchmark: Mixing semi-lay DrugBank texts with technical MedLine abstracts acts as a built-in domain-shift test. The authors attribute the performance gap mainly to differences in writing style.
  • I4. Chained test sets across subtasks: The relation-extraction test set keeps its entity annotations, so it can also serve as extra NER training data. The winning NER team did exactly this.
  • I5. Hidden test gold: Keeping the test labels private and planning a web-based evaluation service guards against overfitting to the test set and keeps the benchmark stable after the competition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 2 Computer Science · 9,086 works Artificial Intelligence · 8,889 works

Cite

Cite as Isabel Segura-Bedmar, Paloma Martínez and María Herrero-Zazo. 2013. SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{segurabedmar2013semeval,
  title = {{SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013)}},
  author = {Isabel Segura-Bedmar, Paloma Martínez and María Herrero-Zazo},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)},
  year = {2013},
  url = {https://aclanthology.org/S13-2056/}
}

References

30 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Airola, S. Pyysalo, J. Bjorne, T. Pahikkala, F. Ginter, and T. Salakoski. 2008. All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning. BMC bioinformatics, 9(Suppl 11):S2. find
  2. JK. Aronson. 2007. Communicating information about drug interactions. British Journal of Clinical Pharmacology, 63(6):637–639, June. find
  3. K. Baxter and I.H. Stockely. 2010. Stockley’s drug interactions.8th ed. London:Pharmaceutical Press. find
  4. J. Björne, J. Heimonen, F. Ginter, A. Airola, T. Pahikkala, and T. Salakoski. 2011. Extracting contextualized complex biological events with graph-based feature sets. Computational Intelligence, 27(4):541–557. find
  5. J. Björne, S. Kaewphan, and T. Salakoski. 2013. UTurku: Drug Named Entity Detection and Drug-drug Interaction Extraction Using SVM Classification and Domain Knowledge. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  6. T. Bobić, J. Fluck, and M. Hofmann-Apitius. 2013. SCAI: Extracting drug-drug interactions using a rich feature vector. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  7. A. Bokharaeian, B.and Dı́az. 2013. NIL UCM: Extracting Drug-Drug interactions from text through combination of sequence and tree kernels. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  8. D. Boring. 1997. The development and adoption of nonproprietary, established, and proprietary names for pharmaceuticals. Drug information journal, 31(3):621–634. find
  9. N. Chinchor and P. Robinson. 1997. Muc-7 named entity task definition. In Proceedings of the 7th Conference on Message Understanding. find
  10. N. Chinchor and B. Sundheim. 1993. Muc-5 evaluation metrics. In Proceedings of the 5th conference on Message understanding, pages 69–78. Association for Computational Linguistics. find
  11. MFM. Chowdhury and A. Lavelli. 2012. Impact of less skewed distributions on efficiency and effectiveness of biomedical relation extraction. In Proceedings of COLING 2012. find
  12. MFM. Chowdhury and A. Lavelli. 2013b. Exploiting the scope of negations and heterogeneous features for relation extraction: Case study drug-drug interaction extraction. In Proceedings of NAACL 2013. find
  13. M.F.M. Chowdhury and A. Lavelli. 2013c. FBK-irst : A Multi-Phase Kernel Based Approach for Drug-Drug Interaction Detection and Classification that Exploits Linguistic Information. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  14. MFM. Chowdhury. 2013a. Improving the Effectiveness of Information Extraction from Biomedical Text. Ph.d. dissertation, University of Trento. find
  15. A. Collazo, A. Ceballo, D Puig, Y. Gutiérrez, J. Abreu, J Pérez, A. Fernández-Orquı́n, A. Montoyo, R. Muñoz, and F. Camara. 2013. UMCC DLSI-(DDI): Semantic and Lexical features for detection and classification Drugs in biomedical texts. In Proceedings of the 7th International Workshop on Semanti… find
  16. C. Giuliano, A. Lavelli, and L. Romano. 2006. Exploiting shallow linguistic information for relation extraction from biomedical literature. In Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics (EACL-2006), pages 401–408. find
  17. T. Grego, F. Pinto, and F.M. Couto. 2013. LASIGE: using Conditional Random Fields and ChEBI ontology. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  18. N.D. Hailu, L.E. Hunter, and K.B. Cohen. 2013. UColorado SOM: Extraction of Drug-Drug Interactions from Biomedical Text using Knowledge-rich and Knowledge-poor Features. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  19. ML. Neves, JM. Carazo, and A. Pascual-Montano. 2009. Extraction of biomedical events using case-based reasoning. In Proceedings of the Workshop on BioNLP: Shared Task, pages 68–76. Association for Computational Linguistics. find
  20. S. Pyysalo, A. Airola, J. Heimonen, J. Bjorne, F. Ginter, and T. Salakoski. 2008. Comparative analysis of five protein-protein interaction corpora. BMC bioinformatics, 9(Suppl 3):S6. find
  21. M. Rastegar-Mojarad, R. D. Boyce, and R. Prasad. 2013. UWM-TRIADS: Classifying Drug-Drug Interactions with Two-Stage SVM and Post-Processing. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  22. T. Rocktäschel, M. Weidlich, and U. Leser. 2012. Chemspot: a hybrid system for chemical named entity recognition. Bioinformatics, 28(12):1633–1640. find
  23. T. Rocktäschel, T. Huber, M. Weidlich, and U. Leser. 2013. WBI-NER: The impact of domain-specific features on the performance of identifying and classifying mentions of drugs. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  24. D. Sanchez-Cisneros and F. Aparicio. 2013. UEM-UC3M: An Ontology-based named entity recognition system for biomedical texts. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  25. D. Sanchez-Cisneros. 2013. UC3M: A kernel-based approach for identify and classify DDIs in biomedical texts. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  26. I. Segura-Bedmar, P. Martı́nez, and C. de Pablo-Sánchez. 2011a. Using a shallow linguistic kernel for drug-drug interaction extraction. Journal of Biomedical Informatics, 44(5):789 – 804. find
  27. I. Segura-Bedmar, P. Martınez, and D. Sánchez-Cisneros. 2011b. The 1st ddiextraction-2011 challenge task: Extraction of drug-drug interactions from biomedical texts. In Proceedings of DDIExtraction-2011 challenge task, pages 1–9. find
  28. P. Thomas, M. Neves, T. Rocktäschel, and U. Leser. 2013. WBI-DDI: Drug-Drug Interaction Extraction using Majority Voting. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  29. E.F. Tjong Kim Sang and F. De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. our card
  30. D.S. Wishart, C. Knox, A.C. Guo, S. Shrivastava, M. Hassanali, P. Stothard, Z. Chang, and J. Woolsey. 2006. Drugbank: a comprehensive resource for in silico drug discovery and exploration. Nucleic acids research, 34(suppl 1):D668–D672. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bedmar-2013-semeval-2013-task-9-extraction/