Elamur Research⌕
☰
home / papers / 2013 / Semantic Annotation of Textual Entailment

Semantic Annotation of Textual Entailment

Paper · 2013 · Assaf Toledo, Stavroula Alexandropoulou, Sophia Katrenko et al.

Summary

RTE corpora only label sentence pairs as entailing or not. They give no account of the reasoning that links a premise (T) to a hypothesis (H), so systems can be ranked by accuracy, but their inference steps cannot be checked against a gold standard. The authors propose a model-theoretic framework for this. Its core is a formally defined interpreted lexicon: words receive semantic types over e and t, plus denotations, some of them fixed logical operators. An informal annotation scheme then tells annotators how to bind T–H material to that lexicon. The work targets restrictive, intersective and appositive modification, because these are frequent in RTE and can be annotated consistently. As a first stage, a narrower scheme was applied to all valid pairs in RTE 1–4. Annotators marked restrictive modification (RMOD), appositives (APP) and conjunction (CONJ), without spelling out the full proof. These phenomena appear in 80.65% of entailments, and annotators agreed 68% of the time on average. The annotated corpus is public and is a step toward a benchmark whose entailments a theorem prover could verify.

Key points

  • The lexicon covers proper names, nouns, verbs, copula (both predicative and equality readings), articles (with the as iota), some, and restrictive versus appositive who. It also covers intersective and restrictive adjectives and adverbs, which are built from helper functions such as AND, I_m, R_m and P_r.
  • Three annotation categories were used: RMOD, APP and CONJ. They were applied only to valid entailment pairs in RTE 1–4, which were found in 80.65% of the entailments.
  • Consistency was measured in four checks of 50–70 jointly annotated pairs each, giving roughly 66.67%, 74.11%, 66.67% and 64.66% identical annotations (about 68% on average).
  • Annotation was done in GATE Developer in two steps:
    • mark the relevant span in T;
    • mark the corresponding span in H that results from the inference.
  • Related logic-based systems showed high precision and very low recall. Bos & Markert reached 76% precision but 5.8% recall on RTE 2. The MacCartney & Manning natural-logic system had a similar profile on RTE 3.

Interesting ideas

  • I1. The lexicon separates content words, which get arbitrary constants, from logical words whose meanings are fixed (iota, existential, conjunction, identity), so only the fixed part drives inference.
  • I2. Intersective modifiers license two inferences ("Dutch man" gives both man and Dutch), while restrictive ones license only the head noun ("short man" is a man, but not short in absolute terms).
  • I3. Appositive relative clauses add a side fact about an entity that is already identified, and they leave the noun's reference unchanged, unlike restrictive clauses.
  • I4. Each annotation links a span in the premise to the span in the hypothesis that it justifies, which gives a traceable evidence chain for every entailment.
  • I5. Annotator disagreements were sorted into missing, wrong scheme or scope, structural ambiguity, inference ambiguity and scheme unclarity, rather than reported only as a single agreement number.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Compiled knowledge could keep a small, hard-coded set of operator words apart from open-class constants, so that inference rules are defined only over the fixed operators.
  • I2 (world): When asserting entity states, the adjective class could decide whether the property is stored as a standalone fact about the entity or only relative to its noun.
  • I3 (frames): Non-restrictive clauses and appositions could be split off as separate assertion frames about the head entity, instead of being folded into its identifying description.
  • I4 (answer): Yes/no and QA readers could output premise-span → hypothesis-span links as their explanation, making each answer checkable step by step.
  • I5 (learn): When witness parsers or teacher tiers disagree on silver data, tagging the cause (attachment ambiguity versus guideline gap) would show whether to fix rules or flag the case as genuinely ambiguous.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Assaf Toledo, Stavroula Alexandropoulou, Sophia Katrenko, Heidi Klockmann, Pepijn Kokke and Yoad Winter. 2013. Semantic Annotation of Textual Entailment. Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{toledo2013semantic,
  title = {{Semantic Annotation of Textual Entailment}},
  author = {Assaf Toledo, Stavroula Alexandropoulou, Sophia Katrenko, Heidi Klockmann, Pepijn Kokke and Yoad Winter},
  booktitle = {Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers},
  year = {2013},
  url = {https://aclanthology.org/W13-0121/}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bar Haim, R., I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, B. Magnini, and I. Szpektor (2006). The second pascal recognising textual entailment challenge. In In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  2. Bentivogli, L., I. Dagan, H. T. Dang, D. Giampiccolo, and B. Magnini (2009). The fifth pascal recognizing textual entailment challenge. Proceedings of TAC 9, 14–24. find
  3. Bos, J., S. Clark, M. Steedman, J. R. Curran, and J. Hockenmaier (2004). Wide-coverage semantic representations from a CCG parser. In Proceedings of the 20th international conference on Computational Linguistics, pp. 12–40. find
  4. Bos, J. and K. Markert (2005). Recognising textual entailment with logical inference. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 628–635.
  5. Cooper, R., D. Crouch, J. Van Eijck, C. Fox, J. Van Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, and K. Konrad (1996). Using the Framework. The Fracas Consortium. find
  6. Cunningham, H., D. Maynard, K. Bontcheva, V. Tablan, N. Aswani, I. Roberts, G. Gorrell, A. Funk, A. Roberts, D. Damljanovic, T. Heitz, M. A. Greenwood, H. Saggion, J. Petrak, Y. Li, and W. Peters (2011). Text Processing with GATE (Version 6). find
  7. Dagan, I., O. Glickman, and B. Magnini (2006). The pascal recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, 177–190. find
  8. Garoufi, K. (2007). Towards a better understanding of applied textual entailment: Annotation and evaluation of the rte-2 dataset. Master’s thesis, Saarland University. find
  9. Giampiccolo, D., H. T. Dang, B. Magnini, I. Dagan, and E. Cabrio (2008). The fourth pascal recognising textual entailment challenge. In In TAC 2008 Proceedings. find
  10. Giampiccolo, D., B. Magnini, I. Dagan, and B. Dolan (2007). The third pascal recognizing textual entailment challenge. In In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, Stroudsburg, PA, USA, pp. 1–9. Association for Computational Linguistics. find
  11. Kamp, H. and U. Reyle (1993). From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse representation theory, Volume 42. Kluwer Academic Dordrecht,, The Netherlands. find
  12. MacCartney, B. and C. D. Manning (2007). Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 193–200. find
  13. Pratt-Hartmann, I. and L. S. Moss (2009). Logics for the relational syllogistic. Review of Symbolic Logic, 647–683. ↗
  14. Stern, A. and I. Dagan (2011). A confidence model for syntactically-motivated entailment proofs. In Proceedings of RANLP 2011. find
  15. Toledo, A., S. Katrenko, S. Alexandropoulou, H. Klockmann, A. Stern, I. Dagan, and Y. Winter (2012). Semantic annotation for textual entailment recognition. In Proceedings of the Eleventh Mexican International Conference on Artificial Intelligence (MICAI). find
  16. Valencia, V. S. (1991). Studies on natural logic and categorial grammar. Ph. D. thesis, University of Amsterdam. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toledo-2013-semantic-annotation-textual-entailment/