Elamur Research⌕
☰
home / papers / 2013 / ClearTK-TimeML: A minimalist approach to TempEval 2013

ClearTK-TimeML: A minimalist approach to TempEval 2013

Paper · 2013 · Steven Bethard

Summary

This paper presents ClearTK-TimeML, a system entered in all English tracks of TempEval 2013: finding events, finding time expressions, and linking them with temporal relations. Built on the ClearTK machine-learning framework, it splits the task into a chain of separate classifiers. Each one uses only a few features that come from tokens, part-of-speech tags or constituency parses. Extents of times and events are handled as BIO chunking. Attributes such as tense, aspect, class and time type get their own classifiers, and time values are delegated to TimeN. The relation models are deliberately narrow. They only cover event–document-time pairs and same-sentence pairs whose syntactic path fits a regular expression, and each predicts only a few relation labels. For every sub-task, a grid search with cross-validation chooses the classifier and its settings. Training sets combined TimeBank, AQUAINT, Silver data and extra relation annotations. The system achieved the top scores on three measures: F1 for temporal relations, accuracy of event tense, and strict F1 for identifying the extents of time expressions. This shows that a lean, selective design can compete with richer systems.

Key points

  • The best relation F1 was 31.0 with system-predicted events and times (TB+BMK), the top score in TempEval. With gold events and times it reached 36.3.
  • Time extent strict F1 was 82.7. Token features included Unicode character-category patterns and a 58-word gazetteer of time words.
  • Of the 14 annotated relation types, each model kept only a small subset:
    • BEFORE/AFTER/INCLUDES for event–document creation time;
    • INCLUDES/IS-INCLUDED for event–time in the same sentence;
    • BEFORE/AFTER for event–event in the same sentence.
  • Adding AQUAINT to TimeBank made time and relation results worse, for example −8.0 in time extent recall. Manual inspection found many annotation errors in that corpus.
  • The Silver data had mixed effects. Time extent precision rose by 8.4, but event tense accuracy fell by 6.6. The BMK verb-clause relations and the PM closure-inferred relations generally raised recall at some cost in precision.

Interesting ideas

  • I1. Prune labels using training errors. Labels the model often confused were folded into NO-RELATION when doing so improved training performance. Giving up hard classes to gain precision could help in any sparse, noisy relation-extraction task.
  • I2. Filter candidates with syntactic paths. Only pairs whose path through the constituency tree matched a regular expression were classified. This simple filter cuts the candidate space and steers the model toward constructions it handles reliably.
  • I3. Compress character categories into a shape feature. Each token is mapped to its sequence of Unicode categories with repeats merged, so "Dec28" becomes "LuLlNd". This captures token shape without language-specific rules and would carry over well to dates, numbers or codes in other languages.
  • I4. Search over classifier families, not only their settings. The learner type and its hyper-parameters were tuned jointly per sub-task with 2-fold cross-validation. This is a cheap way to avoid committing to a single algorithm across a pipeline of many small classifiers.
  • I5. Question whether more data actually helps. Adding high-quality relations still lowered precision. The author suggests this may happen because the reference annotations are incomplete, so correct predictions get scored as errors. This is worth checking whenever gold labels are sparse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States52 works · 49,647 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Steven Bethard. 2013. ClearTK-TimeML: A minimalist approach to TempEval 2013. Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bethard2013cleartk,
  title = {{ClearTK-TimeML: A minimalist approach to TempEval 2013}},
  author = {Steven Bethard},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)},
  year = {2013},
  url = {https://aclanthology.org/S13-2002/}
}

References

10 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bethard and James H. Martin. 2006. Identification of event mentions and their semantic class. In Empirical Methods in Natural Language Processing (EMNLP), page 146154. (Acceptance rate 31%). find
  2. Steven Bethard and James H. Martin. 2007. CU-TMP: temporal relation classification using syntactic and semantic features. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 129–132, Prague, Czech Republic. Association for Computational Linguistics. find
  3. Steven Bethard, James H. Martin, and Sara Klingenstein. 2007. Finding temporal structure in text: Machine learning of syntactic temporal relations. International Journal of Semantic Computing, 01(04):441. find
  4. Hector Llorens, Estela Saquete, and Borja Navarro. 2010. TIPSem (English and Spanish): Evaluating CRFs and semantic roles in TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, page 284291, Uppsala, Sweden, July. Association for Computational Linguistics. find
  5. Hector Llorens, Leon Derczynski, Robert Gaizauskas, and Estela Saquete. 2012. TIMEN: an open temporal expression normalisation resource. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May. European Language Resources Association… find
  6. Philip V. Ogren, Philipp G. Wetzler, and Steven Bethard. 2008. ClearTK: A UIMA toolkit for statistical natural language processing. In Towards Enhanced Interoperability for Large HLT Systems: UIMA for NLP workshop at Language Resources and Evaluation Conference (LREC), 5. find
  7. Naushad UzZaman and James Allen. 2010. TRIPS and TRIOS system for TempEval-2: extracting temporal information from text. In Proceedings of the 5th International Workshop on Semantic Evaluation, page 276283, Uppsala, Sweden, July. Association for Computational Linguistics. find
  8. Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task 1: TempEval-3 evaluating time expressions, events, and temporal relations. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013), in conjun… find
  9. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 task 15: TempEval temporal relation identification. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 75–80, Prague, Czech Republic. Association for Com… find
  10. Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, page 5762, Uppsala, Sweden, July. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bethard-2013-cleartk-timeml-minimalist-approach-tempeval/