Elamur Research⌕
☰
home / papers / 2010 / Expanding textual entailment corpora fromWikipedia using co-training

Expanding textual entailment corpora fromWikipedia using co-training

Paper · 2010 · Fabio Massimo Zanzotto and Marco Pennacchiotti

Summary

Recognizing Textual Entailment (RTE) systems were stuck below roughly 60% accuracy. The RTE challenge corpora were small, about 800 pairs each, and merging them, or adding automatically built sets such as MITRE and LCC, often hurt results because the data were not homogeneous. The authors build the WIKI corpus from Wikipedia revision history. Each pair is an original sentence S1 and its edited version S2. Such pairs are natural, share most of their words, and can be either entailing or not, depending on whether the editor added information or disputed the content. Every edit also comes with the author's comment, which gives a second view of the pair. The authors use co-training with two classifiers, one on the sentence content and one on the comment, to label WIKI pairs starting from RTE-2 training data. An SVM trained on FOSR features finds WIKI about as hard as RTE, unlike a news-based corpus. WIKI is closer to RTE than the news corpus is, and co-training gave early accuracy gains. The approach points to a way of producing large, RTE-compatible training data cheaply.

Key points

  • The corpus is made of (S1, S2) pairs from Wikipedia revisions, where S1 is the text before an edit and S2 the text after it. The editor's comment is kept as an extra signal, for example a note that a person died in a given year.
  • With an SVM on the FOSR feature space, accuracy was 70.73% on WIKI, 94.87% on the news corpus and 60.62% on RTE-2.
    • The very high news score shows that its positive and negative examples are trivially separable.
    • In pilot runs with more news data, accuracy approached 100%.
  • Adding WIKI to RTE-2 training lowered results on the RTE-2 test set. The drop was comparable to adding RTE-1 or RTE-3.
    • RTE-1 + wiki and RTE-3 + wiki both beat RTE-1 and RTE-3 used alone.
  • In the co-training experiment, RTE-2 training served as the labelled set L and unlabelled WIKI pairs as the unlabelled set U. Accuracy rose for some iterations and then levelled off on RTE-3 and declined on RTE-2.
    • This was the only case where the authors saw a gain from data outside the official RTE corpora.
  • For comparison with earlier corpora:
    • MITRE had 100,000 positive-only pairs at about 70% estimated accuracy.
    • LCC had 101,000 positive and 119,000 negative pairs, but its two halves were built by different methods.

Interesting ideas

  • I1. Revision histories supply minimal pairs with near-identical wording where the label can go either way, which removes the shortcut of judging entailment by word overlap.
  • I2. If a classifier separates the classes almost perfectly, that signals a construction artifact, not an easy phenomenon; positives and negatives should come from one shared process.
  • I3. Co-training can use a view that exists only in the unlabelled data, here the edit comment. That view's classifier starts only after the content-view classifier has labelled the first few examples.
  • I4. Homogeneity can be measured by training on a candidate corpus and testing on the target set, then comparing against sibling corpora as reference points.
  • I5. Edits that dispute content give natural contradiction pairs, while edits that add details give natural entailment pairs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Edited-sentence pairs could serve as test items showing that readers decide by meaning, not by shared tokens.
  • I2 (learn): A shallow overlap-only baseline could be run as a negative control on every new silver or test set, flagging sets it solves too easily.
  • I3 (learn): Witness-parser agreement could take a similar side signal, such as source metadata, to vote on silver examples without needing it in gold data.
  • I4 (tree): Before mixing a converted treebank into training, its effect on target-set LAS could be compared with that of an established treebank of the same dialect.
  • I5 (world): Such pairs could test whether the world of a text correctly marks a revised state, such as a changed date, as conflicting with an earlier one rather than extending it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Rome Tor VergataRome · Italy15 works · 1,321 citations🏢Yahoo (United Kingdom)London · United Kingdom3 works · 294 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Fabio Massimo Zanzotto and Marco Pennacchiotti. 2010. Expanding textual entailment corpora fromWikipedia using co-training. Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zanzotto2010expanding,
  title = {{Expanding textual entailment corpora fromWikipedia using co-training}},
  author = {Fabio Massimo Zanzotto and Marco Pennacchiotti},
  booktitle = {Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources},
  year = {2010},
  url = {https://aclanthology.org/W10-3504/}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney. 2002. Bootstrapping. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 360–367, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. find
  2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, and Idan Magnini, Bernardo Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  3. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In COLT: Proceedings of the Conference on Computational Learning Theory. Morgan Kaufmann. ↗
  4. John Burger and Lisa Ferro. 2005. Generating an entailment corpus from news headlines. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 49–54, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  5. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proc. of the 1st NAACL, pages 132–139, Seattle, Washington.
  6. David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129–145. ↗
  7. Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 100–110.
  8. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, and Steve Pulman. 1996. Using the framework. Technical Report LRE 62-051 D-16, The FraCaS Consortium. Technical report. find
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Quionero-Candela et al., editor, LNAI 3944: MLCW 2005, pages 177–190, Milan, Italy. Springer-Verlag. find
  10. Marie-Catherine de Marneffe, Bill MacCartney, Trond Grenager, Daniel Cer, Anna Rafferty, and Christopher D. Manning. 2006. Learning to distinguish valid textual entailments. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  11. A.P. Dempster, N.M. Laird, and D.B. Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, Series B, 39(1):1–38. ↗
  12. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague, June. Association for Computational Linguistics.
  13. Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. Web based probabilistic textual entailment. In Proceedings of the 1st Pascal Challenge Workshop, Southampton, UK. find
  14. David Graff. 2003. English gigaword. find
  15. Andrew Hickl and Jeremy Bensley. 2007. A discourse commitment-based framework for recognizing textual entailment. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 171–176, Prague, June. ACL.
  16. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing textual entailment with LCCs GROUNDHOG system. In Proceedings of the 2nd PASCAL Challenge Workshop on RTE, Venice, Italy. find
  17. N. Japkowicz and S. Stephen. 2002. The class imbalance problem: A systematic study. Intelligent Data Analysis, 6(5). find
  18. Thorsten Joachims. 1999. Making large-scale svm learning practical. In B. Schlkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods-Support Vector Learning. MIT Press. ↗
  19. Frank Keller and Mirella Lapata. 2003. Using the web to obtain frequencies for unseen bigrams. Computational Linguistics, 29(3), September. ↗
  20. M. Kubat and S. Matwin. 1997. Addressing the curse of inbalanced data sets: One-side sampleing. In Proceedings of the 14th International Conference on Machine Learning, pages 179–186. Morgan Kaufmann. find
  21. Alessandro Moschitti and Fabio Massimo Zanzotto. 2007. Fast and effective kernels for relational learning from texts. In Proceedings of the International Conference of Machine Learning (ICML), Corvallis, Oregon. find
  22. Jeremy Nicholson, Nicola Stokes, and Timothy Baldwin. 2006. Detecting entailment using an extended implementation of the basic elements overlap metric. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  23. Christian Siefkes. 2008. An Incrementally Trainable Statistical Approach to Information Extraction. VDM Verlag, Saarbrucken, Germany. find
  24. S. Siegel and Jr. N. J. Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill. find
  25. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on EmNLP, pages 254–263, Honolulu, Hawaii. ACL. Annie Zaenen. submitted. Do give a penny for their thoughts. Journal of Natural Language Engineering.
  26. Fabio Massimo Zanzotto and Lorenzo Dell’Arciprete. 2009. Efficient kernels for sentence pair classification. In Conference on Empirical Methods on Natural Language Processing, pages 91–100, 6-7 August. find
  27. Fabio Massimo Zanzotto and Alessandro Moschitti. 2006. Automatic learning of textual entailments with cross-pair similarities. In Proceedings of the 21st Coling and 44th ACL, pages 401–408, Sydney, Australia, July. find
  28. Fabio Massimo Zanzotto, Marco Pennacchiotti, and Alessandro Moschitti. 2009. A machine learning approach to textual entailment recognition. NATURAL LANGUAGE ENGINEERING, 15-04:551–582. Accepted for pubblica-tion. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zanzotto-2010-expanding-textual-entailment-corpora-fromwikipedia/