Elamur Research⌕
☰
home / papers / 2015 / Semi-Automatic Construction of a Textual Entailment Dataset: Selecting Candidates with Vector Space Models

Semi-Automatic Construction of a Textual Entailment Dataset: Selecting Candidates with Vector Space Models

Paper · 2015 · Erick R. Fonseca and Sandra Maria Aluísio

Summary

Portuguese had almost no labeled data for Recognizing Textual Entailment (RTE). A useful RTE set needs pairs that are not trivial. Negative pairs should still be on related topics, and positive pairs should differ in wording and structure. The authors propose a semi-automatic pipeline for finding such candidates. They take Google News clusters, where several articles cover the same event, and use unsupervised Vector Space Models to choose similar sentences from different documents. They train three count-based models, LSI, LDA and Random Projections, on a large corpus of G1 news. Each sentence is projected into a vector, and pairs are ranked by cosine similarity. A manual check of 100 pairs per model sorted them into useful classes and discard classes. LDA gave the best balance: it had the fewest discarded pairs and many good related-but-non-entailing negatives, which earlier methods struggled to produce. Random Projections leaned towards near-duplicates and positive pairs. The result is a cheap way to bootstrap a gold RTE benchmark for a language with few resources.

Key points

  • The VSMs were trained on about 8 months of G1 news (February to mid-October 2014), roughly 220 thousand articles and 100 million tokens. Candidate pairs come from Google News clusters.
  • Three models were compared: LSI, LDA and Random Projections (RP). Each sentence is treated as a document, and pair similarity is the cosine between the two vectors.
  • Annotation used six classes. Entailment, paraphrase, no relation and overlap are kept as candidates (overlap is still under discussion). Too similar and too different are discarded.
  • Counts per 100 sampled pairs, given first for the four kept classes (entailment, then paraphrase, then no relation, then overlap) and afterwards for the two discard classes (too similar, then too different):
    • LDA: 12/5/35/20/4/24, the fewest discards (28).
    • LSI: 15/3/30/9/0/43.
    • RP: 16/10/14/17/12/31.
  • Code and annotated data are published at nilc.icmc.usp.br/rte-bootstrapper/.

Interesting ideas

  • I1. Several articles about one event naturally supply sentence pairs that are related but phrased independently, and this suits entailment and contrast mining far better than a single document does.
  • I2. The goal is a middle similarity band rather than maximum similarity, with explicit "too similar" and "too different" discard classes, so that a benchmark cannot be solved by surface overlap.
  • I3. Different similarity models skew the output in different directions (LDA towards related negatives, RP towards positives and paraphrases), so the choice of selector can steer the class mix of a dataset.
  • I4. An "overlap" class covers pairs where each sentence has a fact the other lacks, and it is kept apart instead of being forced into yes or no.
  • I5. Hard negatives are pairs on the same topic that do not entail each other, and they are treated as the scarce, valuable part of the dataset.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could gather same-event English news clusters and mine sentence pairs as raw material for testing whether two UD-based meaning graphs describe the same event.
  • I2 (learn): when Mova builds evaluation sets or negative controls, it could filter out near-duplicate and unrelated pairs so that scores reflect real understanding of frames, not word matching.
  • I3 (learn): Mova could deliberately run different candidate selectors to balance positive and negative examples when collecting silver data for its LLM teacher tiers.
  • I4 (world): in the world of a text, Mova could compare the entity/state sets of two sentences and report shared, missing and extra facts, which gives an explainable entailment verdict rather than a binary label.
  • I5 (answer): QA on fables and tales could be tested with distractor statements about the same characters and events that are still not supported, so that the reader must check roles and facts rather than topic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Erick R. Fonseca and Sandra Maria Aluísio. 2015. Semi-Automatic Construction of a Textual Entailment Dataset: Selecting Candidates with Vector Space Models. Proceedings of the 10th Brazilian Symposium in Information and Human Language Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fonseca2015semi,
  title = {{Semi-Automatic Construction of a Textual Entailment Dataset: Selecting Candidates with Vector Space Models}},
  author = {Erick R. Fonseca and Sandra Maria Aluísio},
  booktitle = {Proceedings of the 10th Brazilian Symposium in Information and Human Language Technology},
  year = {2015},
  url = {https://aclanthology.org/W15-5624/}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Androutsopoulos, I. and Malakasiotis, P. (2010). A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research, 38:135–187. ↗
  2. Barzilay, R. and Lee, L. (2003). Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment. In Proceedings of HLT-NAACL 2003, pages 16–23. ↗
  3. Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022. ↗
  4. Dagan, I., Dolan, B., Magnini, B., and Roth, D. (2009). Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. ↗
  5. Dagan, I., Glickman, O., Gan, R., and Magnini, B. (2005). The PAS-CAL Recognising Textual Entailment Challenge. In Proceedings of the PASCAL challenges on Recognizing Textual Entailment. find
  6. Dagan, I., Roth, D., Sammons, M., and Zanzotto, F. M. (2013). Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool. ↗
  7. Dolan, B., Quirk, C., and Brockett, C. (2004). Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. In Proceedings of the 20th International Conference on Computational Linguistics, pages 350–356.
  8. Hickl, A., Bensley, J., Williams, J., Roberts, K., Rink, B., and Shi, Y. (2006). Recognizing Textual Entailment with LCC’s GROUNDHOG System. In Proceedings of the Second PASCAL challenges on Recognizing Textual Entailment, pages 80–85. find
  9. Landauer, T. K. and Dumais, S. T. (1997). A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. Psychological Review, 104(2):211–240. ↗
  10. Le, Q. and Mikolov, T. (2014). Distributed Representations of Sentences and Documents. In Proceedings of the 31 st International Conference on Machine Learning. ↗
  11. Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA. ↗
  12. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., bernardi, R., and Zamparelli, R. (2014). A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223. Semi-Automatic Construction of a Textual Entailment Dataset: Selecting Candidates with Vector Space Models
  13. Mikolov, T., tau Yih, W., and Zweig, G. (2013). Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013). Association for Computational Linguistics.
  14. Pennington, J., Socher, R., and Manning, C. (2014). GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  15. Rodrigo, A., Peñas, A., and Verdejo, F. (2009). Overview of the answer validation exercise 2008. In Evaluating Systems for Multilingual and Multimodal Information Access, volume 5706 of Lecture Notes in Computer Science, pages 296– 313. Springer Berlin Heidelberg. find
  16. Sahlgren, M. (2005). An Introduction to Random Indexing. In Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering. find
  17. Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find
  18. Turney, P. D. and Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141– 188. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fonseca-2015-semi-automatic-construction-textual-entailment/