Elamur Research
☰
home / papers / 2010 / The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text

The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W10-3001.pdf

Summary

The CoNLL-2010 Shared Task introduced uncertainty detection as a competition problem. The goal is to separate factual statements from hedged or speculative ones, which matters for information extraction, for example when mining protein-protein interactions. The organizers defined two tasks. Task1 asks for a binary certain/uncertain decision for each sentence, in two domains: biological scientific articles and Wikipedia, where uncertainty takes the form of "weasel" wording. Task2 asks systems to find hedge cues in biological texts and mark the in-sentence span each cue covers.

The paper describes the annotation guidelines. Cues were marked minimally and scopes maximally, with scope boundaries following syntactic structure. For Wikipedia, only cues were annotated, not scopes. The paper also covers the evaluation metrics and the results, and compares participant systems. Those systems used bag-of-words sentence classifiers, token classifiers, sequence labelers (mostly CRF) and hand-written rules.

Its main contribution is a set of uniform, manually annotated benchmarks plus a survey of approaches, which together give a reference point for later uncertainty detectors.

Key points

  • Two tasks:
    • Task1 is sentence-level uncertainty detection on biological articles and on Wikipedia paragraphs (Task1W).
    • Task2 is detection of hedge cues and their linguistic scope in biological text.
  • Biological annotation:
    • It follows the BioScope corpus conventions.
    • Scopes are contiguous spans based on constituency grammar, extended to the largest syntactic unit that falls under the uncertain reading.
    • Phrases such as "raises the question of" count as complex cues.
  • Task1 approaches:
    • Six systems used a bag-of-words sentence classification setup.
    • Five relied on pure token classification of cues.
    • The rest used sequence labeling, typically Conditional Random Fields.
  • Best biological cue-level result (Table 5): the Tang system reached P/R/F of 81.7 / 81.0 / 81.3. Most entries were in the closed track.
  • Availability: the annotated datasets and tools are freely released at the University of Szeged site and are intended as future benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{farkas2010conll,
  title = {{The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text}},
  author = {Farkas, Richárd and Vincze, Veronika and Móra, György and Csirik, János and Szarvas, György},
  booktitle = {Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task},
  year = {2010},
  url = {https://aclanthology.org/W10-3001/}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eiji Aramaki, Yasuhide Miura, Masatsugu Tonoike, Tomoko Ohkuma, Hiroshi Mashuichi, and Kazuhiko Ohe. 2009. TEXT2TABLE: Medical Text Summarization System Based on Named Entity Recognition and Modality Identification. In Proceedings of the BioNLP 2009 Workshop, pages 185–192, Boulder, Colorado, June. Association for Computational Linguistics. find
  2. Wendy W. Chapman, David Chu, and John N. Dowling. 2007. ConText: An Algorithm for Identifying Contextual Features from Clinical Text. In Proceedings of the ACL Workshop on BioNLP 2007, pages 81–88. find
  3. Mike Conway, Son Doan, and Nigel Collier. 2009. Using Hedges to Enhance a Disease Outbreak Report Text Mining System. In Proceedings of the BioNLP 2009 Workshop, pages 142–143, Boulder, Colorado, June. Association for Computational Linguistics. find
  4. Carol Friedman, Philip O. Alderson, John H. M. Austin, James J. Cimino, and Stephen B. Johnson. 1994. A General Natural-language Text Processor for Clinical Radiology. Journal of the American Medical Informatics Association, 1(2):161–174. find
  5. Viola Ganter and Michael Strube. 2009. Finding Hedges by Chasing Weasels: Hedge Detection Using Wikipedia Tags and Shallow Linguistic Features. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 173–176, Suntec, Singapore, August. Association for Computational Linguistics. find
  6. Feng Ji, Xipeng Qiu, and Xuanjing Huang. 2010. Detecting Hedge Cues and their Scopes with Average Perceptron. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 139–146, Uppsala, Sweden, July. Association for Computational Linguistics. find
  7. Yoshinobu Kano, William A. Baumgartner, Luke McCrohon, Sophia Ananiadou, Kevin B. Cohen, Lawrence Hunter, and Jun’ichi Tsujii. 2009. U-Compare: Share and Compare Text Mining Tools with UIMA. Bioinformatics, 25(15):1997–1998, August. find
  8. Halil Kilicoglu and Sabine Bergler. 2008. Recognizing Speculative Language in Biomedical Research Articles: A Linguistically Motivated Perspective. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, pages 46–53, Columbus, Ohio, June. Association for Computational Linguistics. find
  9. Halil Kilicoglu and Sabine Bergler. 2009. Syntactic Dependency Based Heuristics for Biological Event Extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 119–127, Boulder, Colorado, June. Association for Computational Linguistics. find
  10. Halil Kilicoglu and Sabine Bergler. 2010. A High-Precision Approach to Detecting Hedges and Their Scopes. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 103–110, Uppsala, Sweden, July. Association for Computational Linguistics. find
  11. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on Event Extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 1–9, Boulder, Colorado, June. Association for Computational Linguistics. find
  12. George Lakoff. 1972. Linguistics and natural logic. In The Semantics of Natural Language, pages 545– 665, Dordrecht. Reidel. find
  13. Xinxin Li, Jianping Shen, Xiang Gao, and Xuan Wang. 2010. Exploiting Rich Features for Detecting Hedges and Their Scope. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 36–41, Uppsala, Sweden, July. Association for Computational Linguistics. find
  14. Marc Light, Xin Ying Qiu, and Padmini Srinivasan. 2004. The Language of Bioscience: Facts, Speculations, and Statements in Between. In Proceedings of the HLT-NAACL 2004 Workshop: Biolink 2004, Linking Biological Literature, Ontologies and Databases, pages 17–24. find
  15. Ben Medlock and Ted Briscoe. 2007. Weakly Supervised Learning for Hedge Classification in Scientific Literature. In Proceedings of the ACL, pages 992– 999, Prague, Czech Republic, June. find our card
  16. Roser Morante and Walter Daelemans. 2009. Learning the Scope of Hedge Cues in Biomedical Texts. In Proceedings of the BioNLP 2009 Workshop, pages 28–36, Boulder, Colorado, June. Association for Computational Linguistics. find our card
  17. Roser Morante, Vincent Van Asch, and Walter Daelemans. 2010. Memory-based Resolution of In-sentence Scopes of Hedge Cues. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 48–55, Uppsala, Sweden, July. Association for Computational Linguistics. find
  18. Arzucan Özgür and Dragomir R. Radev. 2009. Detecting Speculations and their Scopes in Scientific Text. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1398–1407, Singapore, August. Association for Computational Linguistics. find
  19. György Szarvas. 2008. Hedge Classification in Biomedical Texts with a Weakly Supervised Selection of Keywords. In Proceedings of ACL-08: HLT, pages 281–289, Columbus, Ohio, June. Association for Computational Linguistics. find
  20. Buzhou Tang, Xiaolong Wang, Xuan Wang, Bo Yuan, and Shixi Fan. 2010. A Cascade Method for Detecting Hedges and their Scope in Natural Language Text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 25–29, Uppsala, Sweden, July. Association for Computational Linguistics. find
  21. Sofie Van Landeghem, Yvan Saeys, Bernard De Baets, and Yves Van de Peer. 2009. Analyzing Text in Search of Bio-molecular Events: A High-precision Machine Learning Framework. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 128–136, Boulder, Colorado, June. Association for Computational Linguistics. find
  22. Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The Bio-Scope Corpus: Biomedical Texts Annotated for Uncertainty, Negation and their Scopes. BMC Bioinformatics, 9(Suppl 11):S9. find
  23. Shaodian Zhang, Hai Zhao, Guodong Zhou, and Bao-liang Lu. 2010. Hedge Detection and Scope Finding by Sequence Labeling with Procedural Feature Selection. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 70–77, Uppsala, Sweden, July. Association for Computational Linguistics. find
  24. Qi Zhao, Chengjie Sun, Bingquan Liu, and Yong Cheng. 2010. Learning to Detect Hedges and their Scope Using CRF. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 64– 69, Uppsala, Sweden, July. Association for Computational Linguistics. find
  25. Huiwei Zhou, Xiaoyan Li, Degen Huang, Zezhong Li, and Yuansheng Yang. 2010. Exploiting Multi-Features to Detect Hedges and Their Scope in Biomedical Texts. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 56– 63, Uppsala, Sweden, July. Association for Computational Linguistics. find