Elamur Research⌕
☰
home / papers / 2013 / Weasels, Hedges and Peacocks: Discourse-level Uncertainty in Wikipedia Articles

Weasels, Hedges and Peacocks: Discourse-level Uncertainty in Wikipedia Articles

Paper · 2013 · Veronika Vincze

Summary

Most earlier work on uncertainty detection treated it as a semantic matter, such as modality or speculation. This paper looks at uncertainty that arises from pragmatics and discourse instead. In these cases the truth of a statement can in principle be settled, but the reader is left asking who said it, how much is meant, or by whose standard. Vincze proposes a classification of such phenomena, which she assumes is language-independent, built around three notions:

  • weasels: missing or vague sources, or underspecified arguments;
  • hedges: fuzziness of quantity or quality, in Lakoff's sense;
  • peacocks: subjective, evaluative wording.

She then manually annotates cues of these types in a Wikipedia corpus that was previously used in the CoNLL-2010 shared task. The re-annotation yields far more cues than the original version and separates cue subtypes. A simple dictionary baseline is evaluated at the cue and sentence levels. The resource is relevant to information extraction, opinion mining, retrieval and genre classification, because each application can pick the uncertainty types it needs.

Key points

  • Weasels are defined broadly. They cover unnamed sources ("some have claimed"), agentless passives, and any relevant argument left underspecified. A citation or an explicit list removes the weasel reading.
  • Hedges come in three subtypes, all defined relative to a contextual reference point:
    • intensifiers, which push a value above it;
    • deintensifiers, which push it below;
    • circumscribers, which mark a range around it. Characters such as the tilde and the hyphen in ranges can also act as cues.
  • Peacocks cover praise words, editorial remarks such as "unfortunately", and contentious labels such as "legendary".
  • The new annotation contains 10,794 cues in 7,336 sentences. The CoNLL-2010 version of the same corpus had 6,725 cues in 4,718 uncertain sentences. In CoNLL-2010, the top F-scores for weasel detection reached 60.2 when evaluated per sentence and only 36.5 when evaluated per cue.
  • The baseline used an 80/20 split. A cue candidate was labelled uncertain if at least half of its training occurrences were uncertain, and ambiguous cues took their most frequent class. Sentence-level scores were significantly higher than cue-level scores (ANOVA, p = 0.0026).

Interesting ideas

  • I1. A coordination test separates the two kinds of uncertainty: if "cue x, but certainly not x" is contradictory, the uncertainty is semantic; otherwise it belongs to discourse. This gives a principled way to split uncertainty labels instead of lumping them together.
  • I2. A weasel is treated as an argument whose identity a reader would reasonably ask about ("who exactly?"), and a citation or an enumeration cancels it. This turns vagueness into a checkable property of argument slots.
  • I3. Hedges are modelled as shifts against a contextual reference point (above, below, or a band around it), and numeric ranges and tildes are treated as hedge markers too. This makes fuzzy quantities representable rather than discarded.
  • I4. The paper argues that lumping several uncertainty types into one complex cue, with no subtypes, probably explains the weak CoNLL-2010 Wikipedia results. Fine-grained labels let each application select what it needs.
  • I5. A frequency-threshold dictionary baseline, scored at both the cue and sentence levels, shows how much a lexicon alone achieves. Sentence-level scores look inflated because one detected cue is enough to flag the whole sentence.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could run this test as an explicit rule to decide whether a hedge cue modifies a proposition's truth value or only its source or precision.
  • I2 (world): entities introduced by bare "some" or "many people", or as agents of agentless passives, could be marked as unresolved sources unless a later span names or lists them, so the answer stage can say who is unknown.
  • I3 (world): measures and values could be stored as intervals tied to a reference point, so "approximately 86,000" or "3-7 days" becomes an interval that maths word problems and QA can reason over explainably.
  • I4 (learn): silver data and evaluation sets could keep separate weasel, hedge and peacock labels, so errors can be attributed to a specific subtype rather than to one blurred class.
  • I5 (learn): this lexicon baseline could serve as a negative control, and cue-level scoring could be reported alongside sentence-level scoring so that multi-cue sentences do not hide misses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SzegedSzeged · Hungary10 works · 1,373 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 1 Discourse analysis · 394 works

Cite

Cite as Veronika Vincze. 2013. Weasels, Hedges and Peacocks: Discourse-level Uncertainty in Wikipedia Articles. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vincze2013weasels,
  title = {{Weasels, Hedges and Peacocks: Discourse-level Uncertainty in Wikipedia Articles}},
  author = {Veronika Vincze},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1044/}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandra Y. Aikhenvald. 2004. Evidentiality. Oxford University Press, Oxford. find
  2. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In Proceedings of LREC’10, Valletta, Malta, May. ELRA. our card
  3. Allan Bell. 1991. The language of the News Media. Blackwell, Oxford. find
  4. Penelope Brown and Stephen C. Levinson. 1987. Politeness: Some universals in language usage. CUP, Cambridge, UK. our card
  5. Gillian Brown and George Yule. 1983. Discourse Analysis. CUP, Cambridge, UK. find
  6. Dan Cristea and Bonnie Webber. 1997. Expectations in incremental discourse processing. In Proceedings of ACL97/EACL97, pages 88–95. Morgan Kaufmann. ↗
  7. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? The pragmatic complexity of veridicality assessment. Computational Linguistics, 38:301–333. our card
  8. Mona T. Diab, Lori S. Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of LAW 2009, pages 68–73. The Association for Computer Linguistics. find
  9. Richárd Farkas, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text. In Proceedings of the CoNLL-2010 Shared Task, pages 1–12, Uppsala, Sweden, July. ACL. our card
  10. Viola Ganter and Michael Strube. 2009. Finding Hedges by Chasing Weasels: Hedge Detection Using Wikipedia Tags and Shallow Linguistic Features. In Proceedings of ACL-IJCNLP 2009, pages 173–176, Suntec, Singapore, August. ACL. our card
  11. Ken Hyland. 1996. Writing without conviction? Hedging in scientific research articles. Applied Linguistics, 17(4):433–454. our card
  12. Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii. 2008. Corpus annotation for mining biomedical events from literature. BMC Bioinformatics, 9(Suppl 10). ↗
  13. George Lakoff. 1973. Hedges: A study in meaning criteria and the logic of fuzzy concepts. Journal of Philosophical Logic, 2(4):458–508. ↗ our card
  14. Ben Medlock and Ted Briscoe. 2007. Weakly Supervised Learning for Hedge Classification in Scientific Literature. In Proceedings of the ACL, pages 992– 999, Prague, Czech Republic, June. our card
  15. Roser Morante and Walter Daelemans. 2011. Annotating Modality and Negation for a Machine Reading Evaluation. In Proceedings of CLEF 2011. find
  16. Raheel Nawaz, Paul Thompson, and Sophia Ananiadou. 2010. Evaluating a meta-knowledge annotation scheme for bio-events. In Proceedings of NESP 2010, pages 69–77, Uppsala, Sweden, July. University of Antwerp. find
  17. Kristen Osenga. 2006. Linguistics and Patent Claim Construction. Rutgers Law Journal, 38(61):61–108. Frank Robert Palmer. find
  18. CUP, Cambridge. 1986. Mood and Modality. find
  19. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of ACL-2013, pages 1650–1659, Sofia, Bulgaria, August. ACL. our card
  20. Vassiliki Rizomilioti. 2006. Exploring epistemic modality in academic discourse using corpora. In Information Technology in Languages for Specific Purposes, pages 53–71. Springer US. ↗
  21. Victoria L. Rubin. 2010. Epistemic modality: From uncertainty to certainty in the context of information seeking as interactions with texts. Information Processing & Management, 46(5):533–540. ↗
  22. Roser Saurı́ and James Pustejovsky. 2009. FactBank: a corpus annotated with event factuality. Language Resources and Evaluation, 43:227–268. find
  23. Roser Saurı́ and James Pustejovsky. 2012. Are you sure that this happened? Assessing the factuality degree of events in text. Computational Linguistics, 38:261–299. our card
  24. György Szarvas, Veronika Vincze, Richárd Farkas, György Móra, and Iryna Gurevych. 2012. Cross-genre and cross-domain detection of semantic uncertainty. Computational Linguistics, 38:335–367. ↗
  25. Özlem Uzuner, Xiaoran Zhang, and Tawanda Sibanda. 2009. Machine Learning and Rule-based Approaches to Assertion Classification. Journal of the American Medical Informatics Association, 16(1):109–115, January. ↗
  26. Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008. The Bio-Scope Corpus: Biomedical Texts Annotated for Uncertainty, Negation and their Scopes. BMC Bioinformatics, 9(Suppl 11):S9. find
  27. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):164–210. ↗
  28. Janyce Wiebe. 2012. Subjectivity word sense disambiguation. In Proceedings of WASSA 2012, page 2, Jeju, Korea, July. ACL. find
  29. Theresa Ann Wilson. 2008. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States. Ph.D. thesis, University of Pittsburgh, Pittsburgh. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vincze-2013-weasels-hedges-peacocks-discourse-level/