Elamur Research⌕
☰
home / papers / 2015 / Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection

Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection

Summary

The paper introduces Context Dependent Evidence Detection (CDED). Given a debate topic, a claim, and related documents, the goal is to locate short, well-formed text segments that back the claim within that topic. The authors separate three evidence types: Study, Expert, and Anecdotal. They built a benchmark from 547 Wikipedia articles covering 58 Debatabase topics. Labeling had two stages: five annotators marked candidates, and a second group of five accepted or rejected them and assigned types. The system is a supervised pipeline:

  • a context-free stage scores how coherent a candidate is and whether it looks like evidence;
  • a context-dependent stage judges how the candidate relates to the claim and topic;
  • a claim selection component sets aside claims that probably have no evidence.

Tests on topics unseen in training show that the models transfer across topics. For many claims, a correct evidence segment of the right type appears among the top 4 candidates. This work helps lay the groundwork for systems that build arguments automatically and support decisions.

Key points

  • Dataset:
    • 39 topics are used for train/test and 19 are held out for feature tuning and auxiliary classifiers.
    • The train/test topics contain 3,057 distinct CDE in 274 articles.
    • The data is heavily skewed: only 31% of claims have any Study-type CDE.
  • Pipeline vs. ablation baselines:
    • Removing the context-free stage (the NCFS baseline) hurts both exact MRR and overlap-based MRR.
    • For Study evidence, the context-dependent component beats simple W2V and BM25 relatedness baselines.
  • Macro-averaged MRR with claim selection (Study / Expert / Anecdotal):
    • Pipeline: 0.25 / 0.34 / 0.04.
    • Basic claim selection (BCS): 0.16 / 0.23 / 0.05.
    • All claims: 0.12 / 0.20 / 0.03.
  • Claim selection coverage: the learned claim selector passes more claims than BCS (34% vs 31% for Study, 52% vs 46% for Expert) while still scoring higher. It performs poorly for Anecdotal, which had only 151 claims with CDE.
  • Remaining difficulties: getting the exact segment boundaries right is still hard, as the gap between exact and overlap MRR shows.

Interesting ideas

  • I1. Two-stage annotation: one group proposes candidates and a separate group votes on them. This raises recall cheaply and then filters by majority, which suits any span-annotation task with subjective labels.
  • I2. Splitting context-free from context-dependent scoring: first ask whether a span is well-formed and evidence-like at all, then check whether it fits this particular claim. The same split could serve citation finding, quote mining, or support retrieval for fact-checking.
  • I3. Scoring relatedness against nearby text: the system compares the claim not only with the candidate but also with its section header and the titles of the citations it references. This cheaply adds document structure to relevance estimation.
  • I4. A claim-level "abstain" classifier: the system first predicts whether a claim has any evidence at all. Tightening its threshold trades coverage for precision in a controlled way, which helps in user-facing tools where wrong suggestions are costly.
  • I5. Reporting exact and overlap MRR together: comparing the two separates "found the right region" from "cut the right span boundaries". This is a useful diagnostic for any span extraction system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1050

@inproceedings{rinott2015show,
  title = {{Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection}},
  author = {Rinott, Ruty and Dankin, Lena and Perez, Carlos Alzate and Khapra, Mitesh M. and Aharoni, Ehud and Slonim, Noam},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1050},
  url = {https://doi.org/10.18653/v1/d15-1050}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund, and Noam Slonim. 2014. A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In Proceedings of the First Workshop on Argumentation Mining, pages… find
  2. Kevin D. Ashley and Vern R. Walker. 2013. Toward constructing evidence-based legal arguments using legal decision documents and machine learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law, ICAIL ’13, pages 176–180, New York, NY, USA. ACM. find
  3. Patrice Bellot, Antoine Doucet, Shlomo Geva, Sairam Gurajada, Jaap Kamps, Gabriella Kazai, Marijn Koolen, Arunav Mishra, Veronique Moriceau, Josiane Mothe, Michael Preminger, Eric SanJuan, Ralf Schenkel, Xavier Tannier, Martin Theobald, Matthew Trappett, and Qiuyue Wang. 2013. Overview of inex 2013.… find
  4. Filip Boltužić and Jan Šnajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49–58, Baltimore, Maryland, June. Association for Computational Linguistics. find
  5. Elena Cabrio and Serena Villata. 2012. Combining textual entailment and argumentation theory for supporting online debates interactions. In ACL (2), pages 208–212. find
  6. Claire Cardie, Nancy Green, Iryna Gurevych, Graeme Hirst, Diane Litman, Smaranda Muresan, Georgios Petasis, Manfred Stede, Marilyn Walker, and Janyce Wiebe, editors. 2015. Proceedings of the Second Workshop on Argumentation Mining. Association for Computational Linguistics, Denver, Colorado, June. find
  7. Marc-Allen Cartright, Henry A. Feild, and James Allan. 2011. Evidence finding using a collection of books. In Proceedings of the 4th ACM Workshop on Online Books, Complementary Social Media and Crowdsourcing, BooksOnline ’11, pages 11–18, New York, NY, USA. ACM. find
  8. I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(04). find
  9. Hoa Trang Dang, Diane Kelly, and Jimmy J. Lin. 2007. Overview of the trec 2007 question answering track. In TREC. find
  10. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA, USA. Associati… find our card
  11. Debanjan Ghosh, Smaranda Muresan, Nina Wacholder, Mark Aakhus, and Matthew Mitsui. 2014. Analyzing argumentative discourse units in online interactions. In Proceedings of the First Workshop on Argumentation Mining, pages 39–48, Baltimore, Maryland, June. Association for Computational Linguistics. find
  12. Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. A probabilistic classification approach for lexical textual entailment. In Manuela M. Veloso and Subbarao Kambhampati, editors, AAAI, pages 1050– 1055. AAAI Press / The MIT Press. find
  13. Nancy Green, Kevin Ashley, Diane Litman, Chris Reed, and Vern Walker, editors. 2014. Proceedings of the First Workshop on Argumentation Mining. Association for Computational Linguistics, Baltimore, Maryland, June. find
  14. Adam Lally, John M. Prager, Michael C. McCord, Branimir Boguraev, Siddharth Patwardhan, James Fan, Paul Fodor, and Jennifer Chu-Carroll. 2012. Question analysis: How watson reads a clue. IBM Journal of Research and Development, 56(3):2. find
  15. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1489– 1500, Dublin, Ireland, August. Dublin City University and A… find our card
  16. Marco Lippi and Paolo Torroni. 2015. Context-independent claim detection for argumentation mining. In Proceedings of the Twenty Fourth International Joint Conference on Artificial Intelligence. AAAI Press. find
  17. Michael C. McCord, William J. Murdock, and Bill K. Boguraev. 2012. Deep parsing in watson. IBM J. Res. Dev., 56(3):264–278, May. find
  18. Michael C. McCord. 1990. Slot grammar: A system for simpler construction of practical natural language grammars. In R. Studer, editor, Natural Language and Logic: Proc. of the International Scientific Symposium, Hamburg, FRG, pages 118–145. Springer, Berlin, Heidelberg. find
  19. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
  20. George A. Miller. 1995. Wordnet: A lexical database for english. COMMUNICATIONS OF THE ACM, 38:39–41. find our card
  21. Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Law (ICAIL 2009),, pages 98–109. ACM. find
  22. Raquel Mochales Palau and Marie-Francine Moens. 2011. Argumentation mining. Artificial Intelligence and Law, 19(1):1–22. find
  23. Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38, Baltimore, Maryland, June. Association for Computational Linguistics. find
  24. Andreas Peldszus. 2014. Towards segment-based recognition of argumentation structure in short texts. In Proceedings of the First Workshop on Argumentation Mining, pages 88–97, Baltimore, Maryland, June. Association for Computational Linguistics. find
  25. Martin F. Porter. 1997. Readings in information retrieval. chapter An Algorithm for Suffix Stripping, pages 313–316. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. find
  26. Richard D. Rieke and Malcolm O. Sillars. 1984. Argumentation and the decision making process. Harper Collins, New York, NY, USA. find
  27. Richard D. Rieke and Malcolm O. Sillars. 2001. Argumentation and Critical Decision Making. Longman. find
  28. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1996. Okapi at trec-3. pages 109–126. find our card
  29. Ariel Rosenfeld and Sarit Kraus. 2015. Providing arguments in discussions based on the prediction of human argumentative behavior. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15). find
  30. Zachary Seech. 2008. Writing Philosophy Papers. Cengage Learning. find
  31. Ramakrishnan Srikant and Rakesh Agrawal. 1996. Mining sequential patterns: Generalizations and performance improvements. In Proceedings of the 5th International Conference on Extending Database Technology: Advances in Database Technology, EDBT ’96, pages 3–17, London, UK, UK. Springer-Verlag. find
  32. Maria Paz Garcia Villalba and Patrick Saint-Dizier. 2012. Some facets of argument mining for opinion analysis. In Computational Models of Argument - Proceedings of COMMA 2012, Vienna, Austria, September 10-12, 2012, pages 23–34. find
  33. Douglas Walton. 2009. Argumentation theory: A very short introduction. In Guillermo Simari and Iyad Rahwan, editors, Argumentation in Artificial Intelligence, pages 1–22. Springer US. find
  34. Simon Wells. 2014. Argument mining: Was ist das? In Proceedings of the 14th International Workshop on Computational Models of Natural Argument, CMNA14. find
  35. Ainur Yessenalina, Yejin Choi, and Claire Cardie. 2010. Automatically generating annotator rationales to improve sentiment classification. In Proceedings of the ACL 2010 Conference Short Papers, ACLShort ’10, pages 336–341, Stroudsburg, PA, USA. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rinott-2015-show-me-your-evidence-automatic/