Elamur Research⌕
☰
home / papers / 2014 / Towards a Semantic Model for Textual Entailment Annotation

Towards a Semantic Model for Textual Entailment Annotation

Paper · 2014 · Assaf Toledo, Stavroula Alexandropoulou, Sophie Chesney et al.

Summary

RTE corpora label each premise–hypothesis pair only as valid or invalid. They say nothing about which linguistic steps license the inference, so systems can be ranked by accuracy but not by whether their reasoning is sound. Toledo and colleagues take a first incremental step toward a gold standard of inferential phenomena. They propose a model-theoretic framework with two parts:

  • a formally defined interpreted lexicon: typed symbols over entities and truth values, plus semantic operators;
  • an informal annotation scheme that tells annotators how to map words and constructions of a T–H pair onto that lexicon.

The work focuses on restrictive, intersective and appositive modification. These are frequent in RTE, can be annotated consistently, and cover many syntactic forms with few concepts. Lexical and world-knowledge steps are handled by shallow alignment rather than modelled. A narrower first-stage scheme (SemAnTE 1.0) marked modification in all valid pairs of RTE 1–4. The targeted constructions appear in 80.65% of entailments, and average inter-annotator agreement was 68%. The annotations are released publicly.

Key points

  • The data are the RTE 1–4 corpora. Judgments of yes-entailment (RTE 1–3) and entailment (RTE 4) count as valid, and everything else counts as invalid.
  • The semantics uses functional types over the basic types e and t. Entailment holds when the truth value of T is less than or equal to that of H in every model. The paper presents entailment itself as a preorder, i.e. reflexive and transitive.
  • Restrictive, intersective and appositive modifiers occur in 80.65% of valid entailments. Cross-annotator agreement averages 68%.
  • The paper reviews earlier systems. Bos and Markert's DRS-plus-theorem-prover approach reached 76% precision but only 5.8% recall on RTE 2. Bentivogli et al. reported 64.4% full agreement and a Dice score of 0.78 on 90 RTE 5 pairs.
  • Some disagreements between annotators are legitimate and are sorted into three categories:
    • Ambig.-Struct.: structural attachment ambiguity;
    • Ambig.-Infer.: more than one way to derive the inference;
    • Ambig.-Scheme: gaps in the scheme, such as modifiees that are not contiguous.

Interesting ideas

  • I1. Each real T–H pair is first simplified to isolate the target phenomenon, then mapped onto a small toy vocabulary (for example, a sentence in which Jan sees a nun who is appositively described as a man and as Dutch), so the proof runs on a tiny controlled vocabulary while the binding records the correspondence.
  • I2. Logic-relevant phenomena (modification) are modelled formally, while lexical steps like "last president → former president" or "sold → possessed" are handed off as explicit shallow alignments instead of being faked.
  • I3. Restrictive, intersective and appositive modification licence dropping or reshaping material from T to reach H, e.g. an apposition yields the bare name.
  • I4. Annotator disagreements are classified into legitimate types (structural ambiguity, alternative inference paths, scheme gaps) instead of being treated as plain errors.
  • I5. Annotation coverage is reported as the share of entailments in which a phenomenon occurs, which grounds the choice of phenomena in corpus frequency.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Abstracting entities and predicates into placeholder symbols before reasoning could let the world builder test modifier inferences independently of lexical knowledge.
  • I2 (answer): Answer derivations could tag each step as either a structural inference or a lexical/world-knowledge bridge, keeping explanations honest about where knowledge was assumed.
  • I3 (frames): Frame building over UD amod/appos/acl:relcl could mark each modifier as restrictive or appositive, so later steps know which deletions preserve truth.
  • I4 (learn): When witness parsers disagree on silver data, sorting disagreements into similar classes would separate true ambiguity (e.g. PP attachment) from parser mistakes before training or rule induction.
  • I5 (learn): Mova could measure what fraction of its test questions depend on each supported construction, to prioritise which phenomena to formalise next.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations🎓University College LondonLondon · United Kingdom59 works · 16,463 citations🎓Utrecht UniversityUtrecht · Netherlands21 works · 1,281 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Assaf Toledo, Stavroula Alexandropoulou, Sophie Chesney, Sophia Katrenko, Heid Klockmann, Pepjin Kokke, Benno Kruit and Yoad Winter. 2014. Towards a Semantic Model for Textual Entailment Annotation. Linguistic Issues in Language Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{toledo2014semantic,
  title = {{Towards a Semantic Model for Textual Entailment Annotation}},
  author = {Assaf Toledo, Stavroula Alexandropoulou, Sophie Chesney, Sophia Katrenko, Heid Klockmann, Pepjin Kokke, Benno Kruit and Yoad Winter},
  booktitle = {Linguistic Issues in Language Technology},
  year = {2014},
  doi = {10.33011/lilt.v9i.1331},
  url = {https://doi.org/10.33011/lilt.v9i.1331}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bar Haim, R., I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, B. Magnini, and I. Szpektor. 2006. The Second PASCAL Recognising Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  2. Bentivogli, L., E. Cabrio, I. Dagan, D. Giampiccolo, M. L. Leggio, and B. Magnini. 2010. Building Textual Entailment Specialized Data Sets: A Methodology for Isolating Linguistic Phenomena Relevant to Inference. In Proceedings of LREC 2010 . find
  3. Bentivogli, L., I. Dagan, H. T. Dang, D. Giampiccolo, and B. Magnini. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge. In Proceedings of TAC , vol. 9, pages 14–24. find
  4. Bos, J., S. Clark, M. Steedman, J. R. Curran, and J. Hockenmaier. 2004. Wide-Coverage Semantic Representations from a CCG Parser. In Proceedings of the 20th international conference on Computational Linguistics, pages 12–40. find
  5. Bos, J. and K. Markert. 2005. Recognising Textual Entailment with Logical Inference. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 628–635.
  6. Cooper, Robin, Dick Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Briscoe, Holger Maier, and Karsten Konrad. 1996. Using the Framework . The Fracas Consortium. find
  7. Cunningham, Hamish, Diana Maynard, Kalina Bontcheva, Valentin Tablan, Niraj Aswani, Ian Roberts, Genevieve Gorrell, Adam Funk, Angus Roberts, Danica Damljanovic, Thomas Heitz, Mark A. Greenwood, Horacio Saggion, Johann Petrak, Yaoyong Li, and Wim Peters. 2011. Text Processing with GATE (Version 6). GATE. find
  8. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment 3944:177–190. find
  9. Dice, L. R. 1945. Measures of the Amount of Ecologic Association Between Species. Ecology 26(3):297–302. ↗
  10. Garoufi, Konstantina. 2007. Towards a Better Understanding of Applied Textual Entailment: Annotation and Evaluation of the RTE-2 Ddataset.. Master’s thesis, Saarland University. find
  11. Giampiccolo, D., H. T. Dang, B. Magnini, I. Dagan, and E. Cabrio. 2008. The Fourth PASCAL Recognising Textual Entailment Challenge. In TAC 2008 Proceedings. find
  12. Giampiccolo, Danilo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The Third PASCAL Recognizing Textual Entailment Challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, pages 1–9. Stroudsburg, PA, USA: Association for Computational Linguistics. find
  13. Kamp, H. and U. Reyle. 1993. From Discourse to Logic: Introduction to Model-Theoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory, vol. 42. Kluwer Academic Dordrecht,, The Netherlands. find
  14. Kundu, G. and D. Roth. 2011. Adapting Text Instead of the Model : An Open Domain Approach. In CoNLL. find
  15. LoBue, Peter and Alexander Yates. 2011. Types of Common-Sense Knowledge Needed for Recognizing Textual Entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2 , HLT ’11, pages Towards a Semantic Model / 149 150 / Toledo et al. 329–334. Stroudsburg, PA, USA: Association for Computational Linguistics.
  16. MacCartney, B. and C. D. Manning. 2007. Natural Logic for Textual Inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193–200. find
  17. Moss, Lawrence S. 2010a. Intersecting Adjectives in Syllogistic Logic. In Proceedings of the 10th and 11th Biennial conference on The mathematics of language, MOL ’07/09, pages 223–237. Berlin, Heidelberg: Springer-Verlag. find
  18. Moss, Lawrence S. 2010b. Syllogistic Logics with Verbs. Journal of Logic and Computation, special issue on papers from Order, Algebra and Logics 20(4):947–967. find
  19. Pratt-Hartmann, Ian. 2003. A Two-Variable Fragment of English. Journal of Logic, Language and Information 12(1):13–45. ↗
  20. Pratt-Hartmann, Ian and Lawrence S. Moss. 2009. Logics for the Relational Syllogistic. The Review of Symbolic Logic 2:647–683. ↗
  21. Sammons, M., V. G. Vydiswaran, and D. Roth. 2010. Ask Not What Textual Entailment Can Do for You... In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1199–1208.
  22. Sánchez Valencia, Victor. 1991. Studies on Natural Logic and Categorial Grammar . Ph.D. thesis, University of Amsterdam. find
  23. Stern, Asher and Ido Dagan. 2011. A Confidence Model for Syntactically-Motivated Entailment Proofs. In Proceedings of RANLP 2011 . find
  24. Toledo, Assaf, Stavroula Alexandropoulou, Sophia Katrenko, Heidi Klockmann, Pepijn Kokke, and Yoad Winter. 2013. Semantic Annotation of Textual Entailment. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, pages 240–251. Potsdam, Germany: Association for Computational Linguistics.
  25. Toledo, Assaf, Sophia Katrenko, Stavroula Alexandropoulou, Heidi Klockmann, Asher Stern, Ido Dagan, and Yoad Winter. 2012. Semantic Annotation for Textual Entailment Recognition. In Proceedings of the Eleventh Mexican International Conference on Artificial Intelligence (MICAI). find
  26. Winter, Y. 2010. Elements of Formal Semantics. Unpublished ms., to appear with Edinburgh University Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toledo-2014-semantic-model-textual-entailment-annotation/