Elamur Research⌕
☰
home / papers / 2017 / SemEval-2017 Task 12: Clinical TempEval

SemEval-2017 Task 12: Clinical TempEval

Paper · 2017 · Steven Bethard, Guergana Savova, Martha Palmer et al.

Summary

Clinical TempEval 2017 asked whether temporal information extraction systems for clinical text still work when the medical condition changes. Earlier editions trained and tested on colon cancer notes. This edition trained on colon cancer and tested on brain cancer, which turns the shared task into a domain adaptation challenge. The data comes from the THYME corpus: Mayo Clinic clinical and pathology notes for 200 colon cancer and 200 brain cancer patients, annotated with a clinical extension of ISO-TimeML. Systems competed on nine sub-tasks covering time expressions, event expressions with their attributes, relations to the document creation time, and narrative container (CONTAINS) relations. There were two settings. In the unsupervised one, systems saw no brain cancer annotations. In the supervised one, they received a small annotated brain cancer sample. Eleven teams took part. Compared with same-domain results from 2016, scores fell by roughly 20 points on most tasks, and the small in-domain sample recovered only a few points. The authors conclude that building clinical timeline extractors that hold up across conditions remains an open problem.

Key points

  • Data: Each THYME corpus was divided into Train/Dev/Test (50/25/25%). The only brain cancer annotations released was Train-10, which covers the first 10 patients and contains 30 documents, 2,557 EVENTs, 350 TIMEX3s and 624 CONTAINS links. Annotations were distributed in Anafora standoff format, and the raw text required a Mayo Clinic data use agreement.
  • Best results: LIMSI-COT led on nearly all tasks. It scored about 0.70 F1 on events without target-domain annotations and about 0.75 with them. On document-time relations it reached 0.51 and 0.59, and on narrative containers about 0.33 and 0.32.
  • Comparison with 2016: In-domain top scores in 2016 were 0.92 F1 for event spans, 0.76 for document-time relations and 0.48 for narrative containers.
  • Why performance dropped: The vocabulary differs sharply between the two domains. Event mentions of terms such as craniotomy, glioma and temozolomide number over 150 in the brain cancer data, yet these terms are absent from the colon cancer data. Formatting also differs: "POST-OP" in all capitals occurs only in the brain cancer notes.
  • Adaptation methods used: Participants relied on simple techniques, such as handling unknown words, giving extra weight to target-domain data, and using generic pre-trained embeddings. More advanced adaptation methods were left unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States73 works · 37,309 citations🎓University of Colorado BoulderBoulder · United States52 works · 49,647 citations🎓Brandeis UniversityWaltham · United States30 works · 5,497 citations🎓University of ArizonaTucson · United States19 works · 10,269 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksBiochemistry, Genetics and Molecular Biology · 116 worksMolecular Biology · 114 works

Cite

Cite as Steven Bethard, Guergana Savova, Martha Palmer and James Pustejovsky. 2017. SemEval-2017 Task 12: Clinical TempEval. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2093

@inproceedings{bethard2017semeval,
  title = {{SemEval-2017 Task 12: Clinical TempEval}},
  author = {Steven Bethard, Guergana Savova, Martha Palmer and James Pustejovsky},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2093},
  url = {https://doi.org/10.18653/v1/s17-2093}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bethard, Leon Derczynski, Guergana Savova, James Pustejovsky, and Marc Verhagen. 2015. SemEval-2015 Task 6: Clinical TempEval. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Association for Computational Linguistics, Denver, Colorado, pages 806–814. ht… find
  2. Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. SemEval-2016 Task 12: Clinical TempEval. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, Californi…
  3. Wei-Te Chen and Will Styler. 2013. Anafora: A web-based general purpose annotation tool. In Proceedings of the 2013 NAACL HLT Demonstration Session. Association for Computational Linguistics, Atlanta, Georgia, pages 14–19. http://www.aclweb.org/anthology/N13-3004. ↗
  4. Po-Yu Huang, Hen-Hsen Huang, Yu-Wun Wang, Ching Huang, and Hsin-Hsi Chen. 2017. NTU-1 at SemEval-2017 Task 12: Detection and classification of temporal events in clinical data with domain adaptation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association… find
  5. Andre Lamurias, Diana Sousa, Sofia Pereira, Luka Clarke, and Francisco M Couto. 2017. ULISBOA at SemEval-2017 Task 12: Extraction and classification of temporal expressions and events. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computatio… find
  6. Artuur Leeuwenberg and Marie-Francine Moens. 2017. KULeuven-LIIR at SemEval-2017 Task 12: Cross-domain temporal information extraction from clinical records. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver,… find
  7. Yu Long, Zhijing Li, Xuan Wang, and Chen Li. 2017. XJNLP at SemEval-2017 Task 12: Clinical temporal information extraction with a hybrid model. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, pages… find
  8. Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2017. GUIR at SemEval-2017 Task 12: A framework for cross-domain clinical temporal information extraction. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, C… find
  9. Sarath P R, Manikandan R, and Yoshiki Niwa. 2017. Hitachi at SemEval-2017 Task 12: System for temporal information extraction from clinical notes. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, pa… find
  10. James Pustejovsky and Amber Stubbs. 2011. Increasing informativeness in temporal annotation. In Proceedings of the 5th Linguistic Annotation Workshop. Association for Computational Linguistics, Portland, Oregon, USA, pages 152–160. http://www.aclweb.org/anthology/W11-0419. ↗
  11. William F. Styler, IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014a. Temporal annotation in the clinical domain. Transactions of the Association for Computational Linguistics 2:143–154…
  12. William F. Styler, IV, Guergana Savova, Martha Palmer, James Pustejovsky, Tim O’Gorman, and Piet C. de Groen. 2014b. THYME annotation guidelines. http://clear.colorado.edu/ compsem/documents/THYMEGuidelines. pdf. ↗
  13. Julien Tourille, Olivier Ferret, Xavier Tannier, and Aurélie Névéol. 2017. LIMSI-COT at SemEval-2017 Task 12: Neural architecture for temporal information extraction from clinical narratives. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for… find
  14. Naushad UzZaman and James Allen. 2011. Temporal evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, pages 351–356. http://www.aclweb. org/anthology/P11-2… ↗
  15. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of… find
  16. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 Task 15: TempEval temporal relation identification. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguist… find
  17. Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 Task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Uppsala, Sweden, pages 57–62. http://www.aclweb.org/ anthology/S10-1010. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bethard-2017-semeval-2017-task-12-clinical/