Elamur Research⌕
☰
home / papers / 2013 / Labeled Alignment for Recognizing Textual Entailment

Labeled Alignment for Recognizing Textual Entailment

Paper · 2013 · Xiaolin Wang, Hai Zhao and Bao-Liang Lu

Summary

Alignment-based approaches to Recognizing Textual Entailment (RTE) borrow their link scheme from machine translation. That scheme records only what premise and hypothesis share, such as paraphrases, synonyms and hyponyms. Unmatched hypothesis words are simply sent to NULL, with no indication of which of them actually blocks the inference. Wang, Zhao and Lu extend this into labeled alignment. Besides ordinary positive links, they add negative links that point to the concrete cause of non-entailment: clashing antonyms, numbers, dates or named entities; sentence-level modifiers like negation or modality that have no counterpart; and new content introduced by the hypothesis. They annotated the simplified Chinese binary-class data of NTCIR-9 RITE1 and NTCIR-10 RITE2 (1,595 pairs) with this scheme and released it. A structured aligner learns both kinds of links. An RBF-kernel SVM then decides entailment from alignment-quality measures plus features describing each link's type. Accuracy rose from 0.7715 to 0.8129 on RITE1 and from 0.6991 to 0.7465 on RITE2, beating both the normal-alignment baseline and every official shared-task entry. The broader lesson is that marking why an inference fails carries as much signal as matching what supports it.

Key points

  • Negative links come in three groups: contradictory expressions, unmatched sentence-level modifiers, and hypothesis novelty.
    • Contradictory expressions include antonyms, number/date/time mismatches, different NEs, mismatched noun-phrase heads and vital modifiers.
    • Unmatched modifiers include negations (not, no, few, without, never, nearly) and modalities.
    • Hypothesis novelty covers numbers or content words with no source in the premise.
  • Data sizes:
    • RITE1: 407 training and 407 test pairs.
    • RITE2: 814 training pairs (all of RITE1 reused) and 781 new test pairs.
    • Share of entailment-positive pairs: 0.649 in RITE1 and 0.596 in RITE2.
  • Three native-speaker undergraduates annotated the data, following Brockett's (2007) methodology. On the RITE1 test set:
    • all three agreed on about 72% of proposed links;
    • two of three agreed on about 24%;
    • three-way disagreement occurred in about 4%.
  • Pipeline components:
    • an aligner modeled on prior work by MacCartney et al. (2008) together with Chambers et al. (2007);
    • BaseSeg CRF segmenter, Stanford factored parser and BaseNER;
    • CiLin and HowNet ontologies;
    • LibSVM classifier tuned by 5-fold cross-validation.
  • Best competing runs were ICRC_HITSZ Run03 at 0.7764 on RITE1 and MIG Run02 at 0.6850 on RITE2. The proposed system scored 0.8129 and 0.7465 respectively.

Interesting ideas

  • I1. Give the aligner explicit "blocking" links, so a mismatch is a located, typed object rather than a vague missing-alignment ratio; this makes a negative verdict inspectable.
  • I2. Treat a sentence-level modifier with no counterpart (never, without, nearly, a modal) as a link to NULL that flips polarity; this captures negation and modality inside the alignment itself rather than through separate ad-hoc features.
  • I3. Separate novelty from contradiction: hypothesis content absent from the premise is a different failure from content that conflicts with it, and labeling them apart sharpens downstream decisions.
  • I4. Generate candidate links from many analysis layers at once (segmented words, syntactic nodes, NEs, synonym/antonym/ontology pairs) and let a learned scorer choose among them, so alignment becomes a fusion of the whole NLP stack.
  • I5. Measure annotation reliability before reconciliation: annotators work independently on a held-out portion, agreement is recorded, and only then are differences resolved.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When a reader rejects a candidate answer, it could cite the specific clashing pair (an entity, number or negation) as its justification instead of reporting a low overlap score.
  • I2 (frames): Frame matching between question and text could record unmatched polarity or modality operators as explicit flip-links on the frame, keeping the reason for a "no" traceable.
  • I3 (world): When checking a claim against the world of a text, keep "not stated" distinct from "contradicted by a stated state or holding." This is effectively an open-world versus closed-world distinction for answers.
  • I4 (tree): Aligning UD subtrees, NE spans and WordNet-related lemma pairs as joint candidates could support tree-level paraphrase matching across sentences.
  • I5 (learn): The same protocol fits silver data, where witness parsers or LLM teacher tiers are compared blind first and only then merged, so disagreement rates stay honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China63 works · 15,047 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Xiaolin Wang, Hai Zhao and Bao-Liang Lu. 2013. Labeled Alignment for Recognizing Textual Entailment. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wang2013labeled,
  title = {{Labeled Alignment for Recognizing Textual Entailment}},
  author = {Xiaolin Wang, Hai Zhao and Bao-Liang Lu},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1069/}
}

References

31 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ion Androutsopoulos and Prodromos Malakasiotis. 2009. A survey of paraphrasing and textual entailment methods. http://arxiv.org/abs/0912.3747. [accessed 10-Jan-2013]. ↗
  2. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of HLT-EMNLP, pages 628–635.
  3. Chris Brockett. 2007. Aligning the RTE 2006 corpus. Microsoft Research Technical Report MSR-TR-2007-77. find
  4. Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  5. Nathanael Chambers, Daniel Cer, Trond Grenager, David Hall, Chloe Kiddon, Bill MacCartney, Marie-Catherine de Marneffe, Daniel Ramage, Eric Yeh, and Christopher D Manning. 2007. Learning alignments and leveraging natural logic. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 165–170. find
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190. find
  7. Zhen Dong Dong and Qiang Dong. 2003. Hownet-a hybrid language and knowledge resource. In Proceedings of International Conference on Natural Language Processing and Knowledge Engineering, pages 820–824. IEEE. find
  8. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing textual entailment with LCC’s GROUND-HOG system. In Proceedings of the Workshop on the Second PASCAL Recognising Textual Entailment Challenge. find
  9. Dan Klein and Christopher D Manning. 2002. Fast exact inference with a factored model for natural language parsing. Advances in neural information processing systems, 15(2003):3–10. find
  10. Milen Kouylekov, Alessio Bosca, and Luca Dini. 2011. EDITS 3.0 at RTE-7. Proceedings of the Seventh PASCAL Recognizing Textual Entailment Challenge. find
  11. Qun Liu and Su Jian Li. 2002. Computation of semantical similarity for phrases based on HowNet (in Chinese). Chinese Computational Linguistics, 7(2):59– 76. find
  12. Hugo Liu and Push Singh. 2004. Conceptnet – a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226. ↗
  13. Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, and Christopher D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of HLT-NAACL, pages 41–48. find
  14. Bill MacCartney, Michel Galley, and Christopher D Manning. 2008. A phrase-based alignment model for natural language inference. In Proceedings of EMNLP’08, pages 802–811. find
  15. Erwin Marsi and Emiel Krahmer. 2005. Classification of semantic relations by humans and machines. In Proceedings of the ACL workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 1–6. find
  16. Yashar Mehdad. 2009. Automatic cost estimation for tree edit distance using particle swarm optimization. In Proceedings of ACL-IJCNLP, pages 289–292. find
  17. Jia Ju Mei, Yi Ming Zhu, and Yun Qi Gao. 1983. TongYiCi CiLin. Shanghai Dictionary Publisher. find
  18. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  19. Sebastia Padó, Michel Galley, Dan Jurafsky, and Chris Manning. 2009. Robust machine translation evaluation with entailment features. In Proceedings of ACL-IJCNLP, pages 297–305. find
  20. Han Ren, Chen Lv, and Donghong Ji. 2011. The WHUTE system in NTCIR-9 RITE task. In Proceedings of NTCIR-9 Workshop Meeting, Tokyo, Japan. find
  21. Miguel Rios and Alexander Gelbukh. 2012. Recognizing textual entailment with a semantic edit distance metric. In 11th Mexican International Conference on Artificial Intelligence,, pages 15–20. IEEE. find
  22. Mark Sammons, VG Vydiswaran, and Dan Roth. 2010. Ask not what textual entailment can do for you... In Proceedings of ACL’10, pages 1199–1208.
  23. Hideki Shima, Hiroshi Kanayama, Cheng-Wei Lee, Chuan-Jie Lin, Teruko Mitamura, Yusuke Miyao, Shuming Shi, and Koichi Takeda. 2011. Overview of NTCIR-9 RITE: Recognizing inference in text. In Proceedings of NTCIR-9 Workshop Meeting, Tokyo, Japan. find
  24. Alexander Volokh and Günter Neumann. 2011. Using MT-based metrics for RTE. In Proceedings of the Seventh PASCAL Recognizing Textual Entailment Challenge. find
  25. Rui Wang and Yi Zhang. 2009. Recognizing textual relatedness with predicate-argument structures. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2, pages 784–792. ACL. find
  26. Yotaro Watanabe, Junta Mizuno, Eric Nichols, Katsuma Narisawa, Keita Nabeshima, Naoaki Okazaki, and Kentaro Inui. 2012. Leveraging diverse lexical resources for textual entailment recognition. ACM Transactions on Asian Language Information Processing, 11(4):18. ↗
  27. Yotaro Watanabe, Yusuke Miyao, Junta Mizuno, Tomohide Shibata, Hiroshi Kanayama, CHeng-Wei Lee, Chuan-Jie Lin, Shuming Shi, Teruko Mitamura, Noriko Kando, Hideki Shima, and Kohichi Takeda. 2013. Overview of the Recognizing Inference in Text (RITE-2) at the NTCIR-10 Workshop. In Proceedings of NTCIR-10. find
  28. Tian Xia. 2007. Research on the computation of semantical similarity for Chinese phrases (in Chinese). Computer Engineering, 33(6):191–194. find
  29. Yaoyun Zhang, Jun Xu, Chenlong Liu, Xiaolong Wang, Ruifeng Xu, Qingcai Chen, Xuan Wang, Yongshuai Hou, and Buzhou Tang. 2011. ICRC HITSZ at RITE: Leveraging multiple classifiers voting for textual entailment recognition. In Proceedings of NTCIR-9 Workshop Meeting, Tokyo, Japan. find
  30. Hai Zhao and Chunyu Kit. 2008. Unsupervised segmentation helps supervised learning of character tagging for word segmentation and named entity recognition. In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing, pages 106–111. find
  31. Hai Zhao, Chang-Ning Huang, and Mu Li. 2006. An improved Chinese word segmentation system with conditional random field. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 162–165. Sydney: July. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2013-labeled-alignment-recognizing-textual-entailment/