Elamur Research⌕
☰
home / papers / 2016 / Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment

Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment

Paper · 2016 · Ngoc Phuoc An Vo and Octavian Popescu

Summary

Semantic relatedness (SR) and textual entailment (TE) are usually handled as separate tasks, even though both judge how the meanings of two sentences relate. Before this work, SICK was the only dataset with both labels on the same pairs, and it targets compositional distributional models with simplified sentences. The authors took the RTE 1-4 corpora from the PASCAL challenges, which already carry TE labels, and added manual SR scores on SICK's 1–5 scale. Annotators worked independently and never saw the entailment labels. The new resource is called RTE+SR 1-4. Their analysis of SICK and RTE+SR shows a strong link between the two kinds of label. They then built a three-step pipeline: estimate SR, use it to decide TE, then use the TE decision to adjust the SR score up or down. For gold pairs in the [2-3] band, this revision raised accuracy by about 20%. The resource matters because it adds realistic text (named entities, dates, numbers, reasoning) to joint SR/TE study.

Key points

  • RTE+SR 1-4: RTE 1-4 pairs were relabelled with SR scores from 1 (unrelated) to 5 (very related), following the SICK annotation guideline. T and H are treated symmetrically as sentence A and sentence B.
  • Annotation setup:
    • The data was split into chunks of about 400 pairs.
    • Three annotators took part, and each chunk was labelled by at least two of them.
    • The TE judgments were hidden from the annotators during annotation.
  • Length rule: RTE texts are unequal in length. If the longer text fully covers the meaning of the shorter one, the pair gets the top SR score, and the shorter text is not penalised.
  • Licensing: Because of RTE-4 licensing, only pair ID and relatedness score are distributed. Users must obtain the original RTE text themselves.
  • Revision result: In the SR-revision step, TE output corrects many errors larger than one point, giving roughly 20% accuracy gain on the [2-3] class. A remaining bias toward the low end of that band was traced to the training data distribution.

Interesting ideas

  • I1. Add a second annotation layer to a corpus that already has labels, and hide the existing labels from annotators so that any correlation found is not an artefact.
  • I2. Use a discrete decision to correct a continuous estimate: score, classify, then shift the score by a learned amount depending on the class.
  • I3. For asymmetric pairs, count full coverage of the shorter text by the longer one as maximal relatedness, rather than penalising the length gap.
  • I4. Release only IDs plus your own labels when the base texts are restricted, so users rebuild the full corpus from their own licensed copy.
  • I5. Check per-band error distributions before and after a correction, not just one overall number, to expose biases inherited from training data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness-parser silver data or LLM-teacher labels for a new layer could be collected blind to the existing layer, so agreement between layers becomes honest evidence.
  • I2 (answer): A graded "how relevant is this passage" score could be nudged after an explicit entailment check, such as whether the fable's world supports the claim.
  • I3 (world): When comparing a question's frame set against a story's world, coverage of the query should be the metric, not overlap size.
  • I4 (learn): Silver annotations over licensed treebanks or tests can be shared as ID-to-label files, keeping the library legally clean.
  • I5 (learn): Evaluation reports could show how errors in each score or difficulty band shift after a repair rule, catching distribution mismatch between train and test.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Fondazione Bruno KesslerTrento · Italy25 works · 43,498 citations🏢XeroxSaint-Denis · France2 works · 4 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Developmental and Educational Psychology · 260 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Ngoc Phuoc An Vo and Octavian Popescu. 2016. Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vo2016corpora,
  title = {{Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment}},
  author = {Ngoc Phuoc An Vo and Octavian Popescu},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/5mt572agpi5m},
  url = {https://doi.org/10.63317/5mt572agpi5m}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 385–393. Association for Computational Linguistics. our card
  2. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. (2006). The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pages 6–4. find
  3. Cooper, R., Crouch, D., Van Eijck, J., Fox, C., Van Genabith, J., Jaspars, J., Kamp, H., Milward, D., Pinkal, M., Poesio, M., et al. (1996). Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  4. Dagan, I., Glickman, O., and Magnini, B. (2006). The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190. Springer. find
  5. Dagan, I., Roth, D., Sammons, M., and Zanzotto, F. M. (2013). Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. ↗
  6. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. (2007). The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. find
  7. Giampiccolo, D., Dang, H. T., Magnini, B., Dagan, I., Cabrio, E., and Dolan, B. (2009). The fourth pascal recognizing textual entailment challenge. In Proceedings of the First Text Analysis Conference (TAC 2008). Citeseer. find
  8. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. (2014a). A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC 2014, Reykjavik (Iceland): ELRA. our card
  9. Marelli, M., Bentivogli, L., Baroni, M., Bernardi, R., Menini, S., and Zamparelli, R. (2014b). Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval-2014. our card
  10. Popescu, O., Cabrio, E., and Magnini, B. (2011). Textual entailment using chain calrifying relationships. In Proceedings of the IJCAI Workshop Learning by Reasoning and its Applications in Intelligent Question-Answering. find
  11. Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiskỳ, T., and Blunsom, P. (2015). Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664. ↗ our card
  12. Vo, N. P. A., Popescu, O., and Caselli, T. (2014). Fbk-tr: Svm for semantic relatedness and corpus patterns for rte. SemEval 2014, page 289. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vo-2016-corpora-learning-mutual-relationship-between/