Elamur Research⌕
☰
home / papers / 2014 / Context Dependent Claim Detection

Context Dependent Claim Detection

Summary

The paper introduces Context Dependent Claim Detection (CDCD). Given a Topic (a short phrase framing a debate) and a few relevant articles, the system must locate Context Dependent Claims (CDCs): short, general statements that directly support or contest that Topic. The authors argue this differs from generic argumentation mining, Argumentative Zoning and Textual Entailment for three reasons. Relevance to the Topic matters. Claim boundaries must be found inside sentences. Claims may be opinions rather than checkable facts.

To support the task, they built a benchmark from Wikipedia using 32 debate motions and a three-stage, five-annotator labeling procedure. They then propose a supervised cascade of three components:

  • selecting sentences that likely contain a claim,
  • determining the exact claim span,
  • ranking the candidates.

The cascade is designed for the extreme class imbalance, since only about 2% of sentences hold a CDC. The authors report that the results are encouraging and that the task is feasible. A manual error analysis of system outputs shows where the system fails. The work matters because it defines a new task and benchmark for decision support and automated argument construction.

Key points

  • Topics are 32 randomly chosen motions from idebate.org's Debatabase, ranging from atheism to US responsibility in the Mexican drug wars.
  • The labeling had three stages, each done independently by 5 in-house annotators: finding Wikipedia articles, proposing candidate CDCs, and confirming or rejecting them. Only candidates accepted by at least 3 of 5 annotators were kept.
  • The resulting dataset has 326 Wikipedia articles and 976 CDCs, about 30 per Topic. Only around 2 in 100 sentences contain a CDC. Pairwise Kappa in the confirmation stage averaged 0.39.
  • Annotators applied five criteria: strength, generality, phrasing, keeping the spirit of the source text, and topic unity.
  • In a manual review of system candidates that did not exactly match the gold labels, roughly 10% were judged acceptable. "Strength failed" was the most frequent rejection reason.

Interesting ideas

  • I1. Funnel of classifiers for rare targets. Splitting detection into sentence filtering, span detection and ranking lets each stage face a less skewed, more manageable problem. This could carry over to any span extraction task where positives are very rare, such as finding specific clauses in Ukrainian legal or trade documents.
  • I2. Functional, task-oriented definition. The authors defined a claim by what it does relative to a Topic (supports or contests it) instead of using an abstract argumentation-theory definition. They found this easier to teach annotators, which suggests choosing definitions for annotation projects that labelers can apply consistently.
  • I3. Separate find / propose / confirm annotation stages with majority voting. Decoupling recall-oriented candidate generation from precision-oriented confirmation is a reusable pattern for building high-quality datasets of elusive phenomena.
  • I4. Categorized error analysis of non-matching outputs. Annotators judged system outputs that did not match the gold labels and assigned each a failure category (e.g., repeats the topic, incoherent, not a sentence). This shows that some "errors" are actually acceptable and reveals which failure types dominate. It is a useful complement to strict-match metrics.
  • I5. Type-specific models as a future direction. The authors suggest training separate classifiers for factual and opinion-like claims, or for different domains, on the assumption that each subtype has its own statistical profile. This idea applies to many heterogeneous classification targets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{levy2014context,
  title = {{Context Dependent Claim Detection}},
  author = {Levy, Ran and Bilu, Yonatan and Hershcovich, Daniel and Aharoni, Ehud and Slonim, Noam},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  url = {https://aclanthology.org/C14-1141/}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund, and Noam Slonim. 2014. A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In Proceedings of the 52nd Annual Meeting of the Association for Co… find
  2. Elena Cabrio and Serena Villata. 2012. Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 208–212, Jeju Island, Korea, July. Assoc… find
  3. I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(04). find
  4. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA, USA. Associati… find our card
  5. A. Freeley and D. Steinberg. 2008. Argumentation and Debate. Cengage Learning. find
  6. Yufan Guo, Anna Korhonen, and Thierry Poibeau. 2011. A weakly-supervised approach to argumentative zoning of scientific documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 273–283, Stroudsburg, PA, USA. Association for Computational Lingu… find
  7. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Knowledge Discovery and Data Mining, pages 168–177. find
  8. Craig Macdonald, Rodrygo L.T. Santos, Iadh Ounis, and Ian Soboroff. 2010. Blog track research at trec. SIGIR Forum, 44(1):58–75, August. find
  9. M. C. McCord, J. W. Murdock, and B. K. Boguraev. 2012. Deep parsing in watson. IBM J. Res. Dev., 56(3):264– 278, May. find
  10. George A. Miller. 1995. Wordnet: A lexical database for english. COMMUNICATIONS OF THE ACM, 38:39–41. find our card
  11. Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Law (ICAIL 2009),, pages 98–109. ACM. find
  12. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1–135. find
  13. Vikas Raykar, Mitesh Khapra, Amrita Saha, Priyanka Agrawal, and Shantanu Godbole. 2014. Subjectivity detection. work in progress. find
  14. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In ACL. find
  15. Ramakrishnan Srikant and Rakesh Agrawal. 1996. Mining sequential patterns: Generalizations and performance improvements. pages 3–17. find
  16. Simone Teufel. 1999. Argumentative zoning: Information extraction from scientific text. Technical report. find
  17. Stephen Toulmin. 1958. The Uses of Argument. Cambridge University Press, Cambridge. find
  18. Adam Wyner, Jodi Schneider, Katie Atkinson, and Trevor J. M. Bench-Capon. 2012. Semi-automated argumentative analysis of online product reviews. In Bart Verheij, Stefan Szeider, and Stefan Woltran, editors, COMMA, volume 245 of Frontiers in Artificial Intelligence and Applications, pages 43–50. IOS… find
  19. Wei Zhang, Clement Yu, and Weiyi Meng. 2007. Opinion retrieval from blogs. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management, CIKM ’07, pages 831–840, New York, NY, USA. ACM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/levy-2014-context-dependent-claim-detection/