Elamur Research⌕
☰
home / papers / 2012 / Cross-Genre and Cross-Domain Detection of Semantic Uncertainty

Cross-Genre and Cross-Domain Detection of Semantic Uncertainty

Paper · 2012 · György Szarvas, Veronika Vincze, Richárd Farkas et al.

Summary

The paper addresses how to detect lexical cues of semantic uncertainty, meaning propositions whose truth cannot be determined given the speaker's mental state, across different genres and domains. Existing annotated resources such as BioScope, FactBank, WikiWeasel and MPQA rest on incompatible notions of uncertainty, which makes it hard to transfer knowledge between them. The authors propose a unified fine-grained subcategorization with definitions, examples and annotation tests, and place earlier annotation efforts within it. They then manually harmonize and relabel three corpora covering scientific/biological, Wikipedia and newswire text. Using shallow features, they compare token-classification and sequence-labeling cue detectors and apply domain adaptation. Cue usage and ambiguity clearly vary by domain, yet even distant source data helps. Domain adaptation substantially lowers the amount of target-domain annotation needed for good performance. This matters for information extraction, retrieval, machine translation and clinical document classification, where systems must separate facts from hedged statements.

Key points

  • The categorization covers four semantic uncertainty types: epistemic, doxastic, investigation and condition. Discourse-level uncertainty and dynamic modality are excluded, and the authors leave dynamic modality for future work.
  • When a new domain has no labeled data, plain cross-training reaches about 60–70% cue-level F-score.
  • Training on 3,000 in-domain sentences performs roughly as well as 1,000 target sentences combined with out-of-domain data and domain adaptation. Once the target set exceeds about 5,000 sentences, adding external data matters little.
  • Filtering source data to sentences containing cues already seen in the target domain gives similar gains. This suggests the benefit comes mainly from better disambiguation of known cues rather than discovery of new ones.
  • Earlier attempts at cross-domain training largely failed. In the CoNLL-2010 shared task, only 3 of more than 20 participants gained anything from out-of-domain data, and only marginally. POS tagging is crucial for resolving ambiguous cues such as "concern" and "may".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓University of StuttgartStuttgart · Germany91 works · 46,376 citations🎓University of SzegedSzeged · Hungary12 works · 1,507 citations🏛️Hungarian Academy of SciencesBudapest · Hungary5 works · 760 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as György Szarvas, Veronika Vincze, Richárd Farkas, György Móra and Iryna Gurevych. 2012. Cross-Genre and Cross-Domain Detection of Semantic Uncertainty. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00098

@article{szarvas2012cross,
  title = {{Cross-Genre and Cross-Domain Detection of Semantic Uncertainty}},
  author = {György Szarvas, Veronika Vincze, Richárd Farkas, György Móra and Iryna Gurevych},
  journal = {Computational Linguistics},
  year = {2012},
  doi = {10.1162/coli\_a\_00098},
  url = {https://doi.org/10.1162/coli_a_00098}
}

References

59 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baker, Kathy, Michael Bloodgood, Mona Diab, Bonnie Dorr, Ed Hovy, Lori Levin, Marjorie McShane, Teruko Mitamura, Sergei Nirenburg, Christine Piatko, Owen Rambow, and Gramm Richardson. 2010. Modality Annotation Guidelines. Technical Report 4, Human Language Technology Center of Excellence, Baltimore,… find
  2. Chapman, Wendy W., David Chu, and John N. Dowling. 2007. ConText: An algorithm for identifying contextual features from clinical text. In Proceedings of the ACL Workshop on BioNLP 2007, pages 81–88, Prague, Czech Republic. find
  3. Clausen, David. 2010. HedgeHunter: A system for hedge detection and uncertainty classification. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 120–125, Uppsala.
  4. Conway, Mike, Son Doan, and Nigel Collier. 2009. Using hedges to enhance a disease outbreak report text mining system. In Proceedings of the BioNLP 2009 Workshop, pages 142–143, Boulder, CO.
  5. Curran, James, Stephen Clark, and Johan Bos. 2007. Linguistically motivated large-scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 33–36, Prague.
  6. Daumé III, Hal. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263, Prague.
  7. Daumé III, Hal and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101–126. find
  8. Falahati, Reza. 2006. The use of hedging across different disciplines and rhetorical sections of research articles. In Proceedings of the 22nd NorthWest Linguistics Conference (NWLC22), pages 99–112, Burnaby. find
  9. Farkas, Richárd and György Szarvas. 2008. Automatic construction of rule-based ICD-9-CM coding systems. BMC Bioinformatics, 9:1–9. find
  10. Farkas, Richárd, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. The CoNLL-2010 Shared Task: Learning to detect hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task…
  11. Fernandes, Eraldo R., Carlos E. M. Crestana, and Ruy L. Milidiú. 2010. Hedge detection using the RelHunter approach. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 64–69, Uppsala.
  12. Friedman, Carol, Philip O. Alderson, John H. M. Austin, James J. Cimino, and Stephen B. Johnson. 1994. A General natural-language text processor for clinical radiology. Journal of the American Medical Informatics Association, 1(2):161–174. find
  13. Ganter, Viola and Michael Strube. 2009. Finding hedges by chasing weasels: Hedge detection using Wikipedia tags and shallow linguistic features. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 173–176, Suntec.
  14. Georgescul, Maria. 2010. A hedgehop over a max-margin framework using hedge cues. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 26–31, Uppsala.
  15. Hyland, Ken. 1994. Hedging in academic writing and EAP textbooks. English for Specific Purposes, 13(3):239–256. find
  16. Hyland, Ken. 1996. Writing without conviction? Hedging in scientific research articles. Applied Linguistics, 17(4):433–454.
  17. Hyland, Ken. 1998. Boosters, hedging and the negotiation of academic knowledge. Text, 18(3):349–382. find
  18. Kiefer, Ferenc. 2005. Lehetőség és szükségszerűség [Possibility and necessity]. Tinta Kiadó, Budapest. find
  19. Kilicoglu, Halil and Sabine Bergler. 2008. Recognizing speculative language in biomedical research articles: A linguistically motivated perspective. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, pages 46–53, Columbus, OH.
  20. Kilicoglu, Halil and Sabine Bergler. 2009. Syntactic dependency based heuristics for biological event extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 119–127, Boulder, CO. find
  21. Kim, Jin-Dong, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on Event Extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 1–9, Boulder, OH.
  22. Kim, Jin-Dong, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from literature. BMC Bioinformatics, 9(Suppl 10). find
  23. Li, Xinxin, Jianping Shen, Xiang Gao, and Xuan Wang. 2010. Exploiting rich features for detecting hedges and their scope. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 78–83, Uppsala.
  24. Light, Marc, Xin Ying Qiu, and Padmini Srinivasan. 2004. The language of bioscience: Facts, speculations, and statements in between. In Proceedings of the HLT-NAACL 2004 Workshop: Biolink 2004, Linking Biological Literature, Ontologies and Databases, pages 17–24, Boston, Massachusetts, USA. find
  25. MacKinlay, Andrew, David Martinez, and Timothy Baldwin. 2009. Biomedical event annotation with CRFs and precision grammars. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing: Shared Task, BioNLP ’09, pages 77–85, Uppsala. find
  26. McCallum, Andrew Kachites. 2002. MALLET: A Machine Learning for Language Toolkit. Available at http://mallet.cs.umass.edu. ↗
  27. Medlock, Ben and Ted Briscoe. 2007. Weakly supervised learning for hedge classification in scientific literature. In Proceedings of the ACL, pages 992–999, Prague.
  28. Morante, Roser and Walter Daelemans. 2009. Learning the scope of hedge cues in biomedical texts. In Proceedings of the BioNLP 2009 Workshop, pages 28–36, Boulder, CO.
  29. Morante, Roser and Walter Daelemans. 2011. Annotating modality and negation for a machine reading evaluation. In Proceedings of CLEF 2011, Amsterdam, Netherlands. find
  30. Morante, Roser, Vincent Van Asch, and Walter Daelemans. 2010. Memory-based resolution of in-sentence scopes of hedge cues. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 40–47, Uppsala, Sweden.
  31. Nawaz, Raheel, Paul Thompson, and Sophia Ananiadou. 2010. Evaluating a meta-knowledge annotation scheme for bio-events. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 69–77, Uppsala. find
  32. Özgür, Arzucan and Dragomir R. Radev. 2009. Detecting speculations and their scopes in scientific text. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1398–1407, Singapore. find
  33. Palmer, Frank Robert. 1979. Modality and the English Modals. Longman, London. find
  34. Palmer, Frank Robert. 1986. Mood and Modality. Cambridge University Press, Cambridge. find
  35. Pan, Sinno Jialin and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. find
  36. Rei, Marek and Ted Briscoe. 2010. Combining manual rules and supervised learning for hedge cue and scope detection. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 56–63, Uppsala.
  37. Rizomilioti, Vassiliki. 2006. Exploring epistemic modality in academic discourse using corpora. In Elisabet Arnó Macia, Antonia Soler Cervera, and Carmen Rueda Ramos, editors, Information Technology in Languages for Specific Purposes, volume 7 of Educational Linguistics. Springer US, New York, page… find
  38. Rubin, Victoria L. 2010. Epistemic modality: From uncertainty to certainty in the context of information seeking as interactions with texts. Information Processing & Management, 46(5):533–540. find
  39. Rubin, Victoria L., Elizabeth D. Liddy, and Noriko Kando. 2005. Certainty identification in texts: Categorization model and manual tagging results. In James G. Shanahan, Yan Qu, and Janyce Wiebe, editors, Computing Attitude and Affect in Text: Theory and Applications (the Information Retrieval Serie… find
  40. Russell, Stuart J. and Peter Norvig. 2010. Artificial Intelligence—A Modern Approach (3rd international edition). Upper Saddle River, NJ: Pearson Education. find
  41. Sánchez, Liliana Mamani, Baoli Li, and Carl Vogel. 2010. Exploiting CCG structures with tree kernels for speculation detection. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 126–131, Uppsala.
  42. Saurı́, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. thesis, Brandeis University, Waltham, MA. find
  43. Saurı́, Roser and James Pustejovsky. 2009. FactBank: A corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. find
  44. Settles, Burr, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning, pages 1–10, Vancouver, Canada. find
  45. Shatkay, Hagit, Fengxia Pan, Andrey Rzhetsky, and W. John Wilbur. 2008. Multi-dimensional classification of biomedical text: Toward automated, practical provision of high-utility text to diverse users. Bioinformatics, 24(18):2086–2093. find
  46. Sun, Chengjie, Lei Lin, Xiaolong Wang, and Yi Guan. 2007. Using maximum entropy model to extract protein-protein interaction information from biomedical literature. In De-Shuang Huang, Donald C. Wunsch, Daniel S. Levine, and Kang-Hyun Jo, editors, Advanced Intelligent Computing Theories and Applicat… find
  47. Szarvas, György. 2008. Hedge classification in biomedical texts with a weakly supervised selection of keywords. In Proceedings of ACL-08: HLT, pages 281–289, Columbus, OH.
  48. Täckström, Oscar, Sumithra Velupillai, Martin Hassel, Gunnar Eriksson, Hercules Dalianis, and Jussi Karlgren. 2010. Uncertainty detection as approximate max-margin sequence labelling. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task,…
  49. Tang, Buzhou, Xiaolong Wang, Xuan Wang, Bo Yuan, and Shixi Fan. 2010. A cascade method for detecting hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 13–17, Uppsala.
  50. Thompson, Paul, Giulia Venturi, John McNaught, Simonetta Montemagni, and Sophia Ananiadou. 2008. Categorising modality in biomedical texts. In Proceedings of the LREC 2008 Workshop on Building and Evaluating Resources for Biomedical Text Mining, pages 27–34, Marrakech, Morocco. find
  51. Tjong Kim Sang, Erik. 2010. A baseline approach for detecting sentences containing uncertainty. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 148–150, Uppsala. find
  52. Uzuner, Özlem, Xiaoran Zhang, and Tawanda Sibanda. 2009. Machine learning and rule-based approaches to assertion classification. Journal of the American Medical Informatics Association, 16(1):109–115. find
  53. Van Asch, Vincent and Walter Daelemans. 2010. Using domain similarity for performance estimation. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 31–36, Uppsala. find
  54. Van Landeghem, Sofie, Yvan Saeys, Bernard De Baets, and Yves Van de Peer. 2009. Analyzing text in search of bio-molecular events: A high-precision machine learning framework. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 128–136, Boulder, CO. find
  55. Velldal, Erik. 2010. Detecting uncertainty in biomedical literature: A simple disambiguation approach using sparse random indexing. In Proceedings of SMBM 2010, pages 75–83, Cambridge. find
  56. Velldal, Erik, Lilja Øvrelid, and Stephan Oepen. 2010. Resolving speculation: MaxEnt cue classification and dependency-based scope rules. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 48–55, Uppsala. find
  57. Vincze, Veronika, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The BioScope Corpus: Biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics, 9(Suppl 11):S9. find
  58. Wilson, Theresa Ann. 2008. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States. Ph.D. thesis, University of Pittsburgh, PA. find
  59. Zhang, Shaodian, Hai Zhao, Guodong Zhou, and Bao-Liang Lu. 2010. Hedge detection and scope finding by sequence labeling with normalized feature selection. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL-2010): Shared Task, pages 92–99, Uppsala. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/szarvas-2012-cross-genre-cross-domain-detection/