Elamur Research⌕
☰
home / papers / 2007 / The Third PASCAL Recognizing Textual Entailment Challenge

The Third PASCAL Recognizing Textual Entailment Challenge

Paper · 2007 · Danilo Giampiccolo, Bernardo Magnini, Ido Dagan et al.

Summary

The paper summarizes the Third PASCAL Recognizing Textual Entailment Challenge (RTE-3). Participants had to determine whether the meaning of a short hypothesis h can be inferred from a text t. Such semantic inference is needed by many NLP applications, including QA, IE, summarization and IR, so the organizers made it a separate task. They hope that a generic entailment "engine" will eventually emerge that can be plugged in as a standard module. As before, the corpus contains 1600 t-h pairs, created manually in four application settings. This time, longer texts of up to a paragraph were added for the first time. A shared RTE Resource Pool was also introduced for exchanging tools and knowledge. NIST launched a pilot task with three-way labeling and justifications of decisions. In total, 26 teams submitted 44 runs, and the results mostly exceeded those of previous rounds. The work shows that the data creation methodology is reliable. At the same time, the authors acknowledge weaknesses: progress in knowledge acquisition is small, and the distribution of examples in the corpus is largely arbitrary.

Key points

  • The RTE-3 corpus has 1600 pairs, split equally into development and test. Each of the IE, IR, QA and SUM settings accounts for 200 pairs in each part. Negative examples are slightly more numerous: 51.50% and 51.25%.
  • Texts longer than 270 bytes are marked as L. About 17% of the test part are such texts. Accuracy on them (58.72%) is only slightly lower than on short ones (61.93%).
  • The pairs were built from the outputs of real systems and from publicly available data: ACE-2004, MUC-4, TREC, CLEF, WikiNews, Wikipedia.
  • Average accuracy depends markedly on the setting: QA — 0.71, IR — 0.66, SUM — 0.58, IE — 0.52.
  • The NIST pilot task required YES/NO/UNKNOWN decisions, that is, distinguishing contradiction from unknown, as well as justifying the decisions. Among the new approaches, the authors highlight Hickl's commitment-based model, Bar-Haim's proof system and Stanford's use of Natural Logic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Bar-Ilan UniversityRamat Gan · Israel141 works · 23,225 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language understanding · 51 works

Cite

Cite as Danilo Giampiccolo, Bernardo Magnini, Ido Dagan and Bill Dolan. 2007. The Third PASCAL Recognizing Textual Entailment Challenge. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{giampiccolo2007third,
  title = {{The Third PASCAL Recognizing Textual Entailment Challenge}},
  author = {Danilo Giampiccolo, Bernardo Magnini, Ido Dagan and Bill Dolan},
  booktitle = {Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing},
  year = {2007},
  doi = {10.3115/1654536.1654538},
  url = {https://doi.org/10.3115/1654536.1654538}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini and Idan Szpektor. 2006. The Second PASCAL Recognizing Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognizing Textual Entailment, Venice, Italy. find
  2. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognizing Textual Entailment Challenge. In Quiñonero-Candela et al., editors, MLCW 2005, LNAI Volume 3944, pages 177-190. Springer-Verlag. find
  3. J. R. Landis and G. G. Koch. 1997. The measurements of observer agreement for categorical data. Biometrics, 33:159–174. find
  4. Rebecca Passonneau, Ani Nenkova., Kathleen McKeown, and Sergey Sigleman. 2005. Applying the pyramid method in DUC 2005. In Proceedings of the Document Understanding Conference (DUC 05), Vancouver, B.C., Canada. ↗
  5. Ellen M. Voorhees and Donna Harman. 1999. Overview of the seventh text retrieval conference. In Proceedings of the Seventh Text Retrieval Conference (TREC-7). NIST Special Publication. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/