Elamur Research⌕
☰
home / papers / 2014 / Temporal Annotation in the Clinical Domain

Temporal Annotation in the Clinical Domain

Summary

Clinical narratives in electronic health records hold rich temporal information, but much of it is left implicit. Doctors use shorthand and assume shared medical knowledge, so ordering events into a patient timeline is hard for both annotators and systems. The authors present THYME-TimeML, a clinical adaptation and extension of ISO-TimeML. They used it to annotate the THYME corpus of de-identified Mayo Clinic notes on brain and colon cancer.

The guidelines take a broader view of what counts as an event: anything that belongs on a patient's care timeline. They also add document- and section-level time anchors and organize relations around narrative containers. Annotators mark only orderings signaled by language and leave orderings inferable from domain knowledge to later reasoning.

Inter-annotator agreement was solid for events and time expressions. Agreement on temporal links was clearly better than in the earlier i2b2 data. A top TempEval 2013 system lost considerable accuracy on this data, which shows a real domain gap. The corpus is publicly released and was proposed for a SemEval 2015 task.

Key points

  • The THYME corpus contains 1,254 de-identified Mayo Clinic notes from two oncology areas (brain cancer and colon cancer), covering clinical notes and pathology reports.
  • THYME-TimeML adds a DOCTIME anchor for each note and a SECTIONTIME for sections updated later. A simplified version of the guidelines underpinned the 2012 i2b2 temporal relation challenge.
  • CONTAINS makes up 66% of TLINKs and OVERLAP 14.6%. Timelines are therefore built mostly from containment links rather than BEFORE-style ordering.
  • Agreement F1 scores were 0.8038 for EVENTs (i2b2: 0.87), 0.8047 for TIMEX3s (i2b2: 0.89), and 0.5012 for all links (i2b2 TLINK: 0.39).
  • An SVM with constituency tree kernels reached F1 0.737 on EVENT–TIMEX3 narrative container detection. The data and guidelines are available at thyme.healthnlp.org.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

108 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00172

@article{iv2014temporal,
  title = {{Temporal Annotation in the Clinical Domain}},
  author = {IV, William F. Styler and Bethard, Steven and Finan, Sean and Palmer, Martha and Pradhan, Sameer and Groen, Piet C de and Erickson, Brad and Miller, Timothy and Lin, Chen and Savova, Guergana and Pustejovsky, James},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00172},
  url = {https://doi.org/10.1162/tacl_a_00172}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F Allen. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843. find
  2. Emmon Bach. 1986. The algebra of events. Linguistics and philosophy, 9(1):5–16. find
  3. Steven Bethard. 2013. Cleartk-timeml: A minimalist approach to tempeval 2013. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 10–14, Atlanta, Georgia, USA, June. Associa… find
  4. Olivier Bodenreider. 2004. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic acids research, 32(Database issue):D267–D270, January. find
  5. Philip Bramsen, Pawan Deshpande, Yoong Keok Lee, and Regina Barzilay. 2006. Finding temporal order in discharge summaries. In AMIA Annual Symposium Proceedings, volume 2006, page 81. find
  6. American Medical Informatics Association. Carlo Combi, Yuval Shahar, et al. 1997. Temporal reasoning and temporal data maintenance in medicine: issues and challenges. Computers in biology and medicine, 27(5):353–368. find
  7. Robert H Dolin. 1995. Modeling the temporal complexities of symptoms. Journal of the American Medical Informatics Association, 2(5):323–331. find
  8. George Hripcsak, Nicholas D Soulakis, Li Li, Frances P Morrison, Albert M Lai, Carol Friedman, Neil S Cal-man, and Farzad Mostashari. 2009. Syndromic surveillance using ambulatory electronic health records. Journal of the American Medical Informatics Association, 16(3):354–361. find
  9. Ann K Irvine, Stephanie W Haas, and Tessa Sullivan. 2008. Tn-ties: A system for extracting temporal information from emergency department triage notes. In AMIA Annual Symposium proceedings, volume 2008, page 328. find
  10. American Medical Informatics Association. Elpida T Keravnou. 1997. Temporal abstraction of medical data: Deriving periodicity. In Intelligent Data Analysis in Medicine and Pharmacology, pages 61–79. find
  11. Springer. Klaus H. Krippendorff. 2012. Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc, third edition edition, April. Chen Lin, Elizabeth Karlson, Dmitriy Dligach, Monica Ramirez, Timothy Miller, Huan Mo, Natalie Braggs, Andrew Cagan, Joshua Denny, and Guergana. Savova.… find
  12. John McCarthy. 2002. Actions and other events in situation calculus. In Proceedings of the International conference on Principles of Knowledge Representation and Reasoning, pages 615–628. find
  13. Morgan Kaufmann Publishers; 1998. Stéphane M Meystre, Guergana K Savova, Karin C Kipper-Schuler, John F Hurdle, et al. 2008. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform, 35:128–44. find
  14. Timothy Miller, Steven Bethard, Dmitriy Dligach, Sameer Pradhan, Chen Lin, and Guergana Savova. 2013. Discovering temporal narrative containers in clinical text. In Proceedings of the 2013 Workshop on Biomedical Natural Langua ge Processing, pages 18–26, Sofia, Bulgaria, August. Association for Comp… find
  15. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The penn discourse treebank. In In Proceedings of LREC 2004. find our card
  16. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the gnome corpus. In In Proceedings of the ACL Workshop on Discourse Annotation. find
  17. James Pustejovsky and Amber Stubbs. 2011. Increasing informativeness in temporal annotation. In Proceedings of the 5th Linguistic Annotation Workshop, pages 152– 160. Association for Computational Linguistics. find our card
  18. James Pustejovsky, Robert Knippen, Jessica Littman, and Roser Sauri. 2005. Temporal and event information in natural language text. Language Resources and Evaluation, 39(2-3):123–164. find
  19. James Pustejovsky, Kiyong Lee, Harry Bunt, and Laurent Romary. 2010. Iso-timeml: An international standard for semantic annotation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta. find
  20. Angus Roberts, Robert Gaizauskas, Mark Hepple, George Demetriou, Yikun Guo, and Ian Roberts. 2009. Building a semantically annotated corpus of clinical texts. Journal of biomedical informatics, 42(5):950–966. find
  21. Guergana Savova, Steven Bethard, Will Styler, James Martin, Martha Palmer, James Masanz, and Wayne Ward. 2009. Towards temporal relation discovery from the clinical narrative. In AMIA Annual Symposium Proceedings, volume 2009, page 568. find
  22. American Medical Informatics Association. Tessa Sullivan, Ann Irvine, and Stephanie W Haas. 2008. It’s all relative: usage of relative temporal expressions in triage notes. Proceedings of the American Society for Information Science and Technology, 45(1):1–8. find
  23. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013a. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association. find
  24. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013b. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association, 20(5):806–813. find
  25. Alexander Turchin, Maria Shubina, Eugene Breydo, Merri L Pendergrass, and Jonathan S Einbinder. 2009. Comparison of information content of structured and narrative text data sources on the example of medication intensification. Journal of the American Medical Informatics Association, 16(3):362–370. find
  26. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of… find
  27. Marc Verhagen. 2005. Temporal Closure in an Annotation Environment. Language Resources and Evaluation, 39(2):211–241. find
  28. Veronika Vincze, Gyrgy Szarvas, Richrd Farkas, Gyrgy Mra, and Jnos Csirik. 2008. The bioscope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics, 9(Suppl 11):1– 9. find
  29. Ying Zhao, George Karypis, and Usama M. Fayyad. 2005. Hierarchical clustering algorithms for document datasets. Data Mining and Knowledge Discovery, 10:141–168. find
  30. Jiaping Zheng, Wendy W Chapman, Rebecca S Crowley, and Guergana K Savova. 2011. Coreference resolution: A review of general methodologies and applications in the clinical domain. Journal of biomedical informatics, 44(6):1113–1122. find
  31. Li Zhou, Simon Parsons, and George Hripcsak. 2008. The evaluation of a temporal reasoning system in processing clinical discharge summaries. Journal of the American Medical Informatics Association, 15(1):99–106. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iv-2014-temporal-annotation-clinical-domain/