Elamur Research⌕
☰
home / papers / 2008 / Finding Contradictions in Text

Finding Contradictions in Text

Summary

The paper addresses automatic detection of conflicting statements across texts. The authors argue this task differs from textual entailment and is harder. They replace the strict logical notion of contradiction with a practical one: two sentences contradict when they are highly unlikely to be true together, provided they describe the same event. Under this definition they annotated the RTE1–RTE3 datasets for contradiction. NIST later used their guidelines to label RTE3 test for the RTE3 Pilot three-way task. Analysing these data and a corpus of naturally occurring contradictions, they propose a typology with two groups:

  • "Easy" cases: antonymy, negation, and number/date mismatch.
  • Harder cases: factive/modal, structural, lexical, and world-knowledge conflicts.

Their detector includes an event coreference component based on topicality, and they give the first per-type performance breakdown. The system handles negation and antonymy reasonably well but rarely catches lexical and world-knowledge contradictions. It also degrades on test data more than typical entailment systems do, yet it still clearly beats the average RTE3 Pilot entry. The study shows which contradiction types are currently solvable and where effort is needed.

Key points

  • Contradictions are rare even in inference-oriented data: for example, 149 of 800 pairs in RTE1 test and 72 of 800 in RTE3 test.
  • The type distribution differs sharply between the RTE3 dev set and the "real" corpus. Numeric mismatches are 8.8% in RTE3 dev versus 29.0% in the real corpus, and world-knowledge cases are 27.5% versus 13.0%. The easy category covers about a third of RTE contradictions but about half of the real ones.
  • Recall by type on RTE3 test:
    • antonym 42.9% (3/7)
    • negation 60.0% (3/5)
    • lexical 0% (0/12)
    • world knowledge 8.3% (1/12)
  • Further results:
    • 63% accuracy on the negation test set
    • 78.9% precision for single-word antonymy
  • Earlier work by Harabagiu et al. (2006) scored 75.63% on the negation portion of the LCC data and 62.55% on its paraphrase portion, though it dealt only with those two construction types.
  • Structural features were the main source of false alarms. They caused 36 false positives on RTE3 test, more than 75% of the precision errors. The heuristic "same subject, different object" signalled a real contradiction only 55% of the time on the development sets.

Interesting ideas

  • I1. Define the target by human plausibility, not logic. Treating "very unlikely to co-occur" as contradiction makes annotation align with intuition and with practical use cases such as flagging inconsistent reports. Any fact-checking or data-consistency project could adopt a similar operational definition.
  • I2. Make same-event detection a gate. Checking whether two texts describe the same event (via event coreference) before judging conflict prevents false alarms between unrelated statements. This filter design transfers to deduplication or cross-source verification pipelines.
  • I3. Break evaluation down by phenomenon. Reporting recall per contradiction type exposes where a system actually works, rather than hiding failures behind one aggregate score. The same practice helps diagnose any classifier evaluated on mixed phenomena.
  • I4. Compare benchmark and real-world distributions. The relative frequency of contradiction types in RTE differed strongly from naturally occurring data. Checking this before choosing what to optimize keeps development aligned with real use.
  • I5. Mismatch is not conflict. The Nike example compares a profit figure with an orders figure, so differing values are not a real conflict. It warns that surface disagreement needs contextual grounding, a useful caution for numeric consistency checks in trade or statistical data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marneffe2008finding,
  title = {{Finding Contradictions in Text}},
  author = {Marneffe, Marie-Catherine de and Rafferty, Anna N. and Manning, Christopher D.},
  booktitle = {Proceedings of ACL-08: HLT},
  year = {2008},
  url = {https://aclanthology.org/P08-1118/}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  2. Nathanael Chambers, Daniel Cer, Trond Grenager, David Hall, Chloe Kiddon, Bill MacCartney, Marie-Catherine de Marneffe, Daniel Ramage, Eric Yeh, and Christopher D. Manning. 2007. Learning alignments and leveraging natural logic. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Par… find
  3. Timothy Chklovski and Patrick Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In Proceedings of EMNLP-04. find
  4. Cleo Condoravdi, Dick Crouch, Valeria de Pavia, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. Workshop on Text Meaning (2003 May 31). find
  5. Koby Crammer and Yoram Singer. 2001. Ultraconservative online algorithms for multiclass problems. In Proceedings of COLT-2001. find
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Quinonero-Candela et al., editor, MLCW 2005, LNAI Volume 3944, pages 177–190. Springer-Verlag. find
  7. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-06). find our card
  8. Christiane Fellbaum. 1998. WordNet: an electronic lexical database. MIT Press. find
  9. Jan Firbas. 1971. On the concept of communicative dynamism in the theory of functional sentence perspective. Brno Studies in English, 7:23–47. find
  10. Danilo Giampiccolo, Ido Dagan, Bernardo Magnini, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find
  11. Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast, and contradiction in text processing. In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI-06). find
  12. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing textual entailment with LCC’s GROUNDHOG system. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  13. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. 1997. Event coreference for information extraction. In Proceedings of the Workshop on Operational Factors in Pratical, Robust Anaphora Resolution for Unrestricted Texts, 35th ACL meeting. find our card
  14. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics. find our card
  15. Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, and Christopher D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of the North American Association of Computational Linguistics (NAACL-06). find
  16. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find our card
  17. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of ICoS-5. find our card
  18. Olivia Sanchez-Graillet and Massimo Poesio. 2007. Discovering contradiction protein-protein interactions in text. In Proceedings of BioNLP 2007: Biological, translational, and clinical language processing. find
  19. Lucy Vanderwende, Arul Menezes, and Rion Snow. 2006. Microsoft research at rte-2: Syntactic contributions in the entailment task: an implementation. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  20. Ellen Voorhees. 2008. Contradictions and justifications: Extensions to the textual entailment task. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics. find
  21. Annie Zaenen, Lauri Karttunen, and Richard S. Crouch. 2005. Local textual inference: can it be defined or circumscribed? In ACL 2005 Workshop on Empirical Modeling of Semantic Equivalence and Entailment. find
  22. Fabio Massimo Zanzotto, Marco Pennacchiotti, and Alessandro Moschitti. 2007. Shallow semantics in fast textual entailment rule learners. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marneffe-2008-finding-contradictions-text/