Elamur Research⌕
☰
home / papers / 2017 / Recognizing Textual Entailment in Twitter Using Word Embeddings

Recognizing Textual Entailment in Twitter Using Word Embeddings

Paper · 2017 · Octavia-Maria Şulea

Summary

The paper asks whether neural classifiers and word embeddings, which work well on large RTE corpora such as SNLI, can also judge relations between very short, noisy Twitter texts. Each tweet pair must be labelled as entailment, contradiction, or unknown. The author uses a hand-annotated set of about 5000 tweet pairs (Lendvai et al., 2016) covering four news events. Tweets are represented with Google's pretrained word2vec vectors. Pairs are encoded either through distance features (cosine and word mover's distance) or by concatenating summed vectors, with or without a zero separator. Classifiers include an MLP in Keras/Theano, logistic regression and a linear SVM. Under an ordinary train-test split, scores approach the state of the art, with an averaged F1 around 0.90. When an entire event is held out, which is closer to real use, performance falls to roughly 0.45, not far above a dummy baseline. The main lesson is that strong in-domain numbers can hide poor transfer to unseen topics.

Key points

  • Data: about 5000 tweet pairs on four events (Charlie Hebdo shooting, Ottawa shooting, Sydney Siege, German Wings crash). Unknown is the largest class and contradiction the smallest. Contradictions can be as subtle as a different victim count or location.
  • Features: 300-dim word2vec sums per tweet. These are concatenated into a 600-dim vector, or into a 900-dim vector with a 300-dim zero separator block. A separate setup uses only a two-column feature of cosine plus word mover's distance (computed via pyemd).
  • Training settings: the MLP was trained with batch size 50 for 100 epochs.
  • Train-test split: the best model reached F1 ≈ 0.90 averaged and ≈ 0.82 on contradiction. Logistic regression reached ≈ 0.78 and ≈ 0.61, and the baseline 0.33.
  • Event-based 4-fold CV:
    • The 900-dim vectors give averaged F1 ≈ 0.45 for both MLP and LR, against 0.33 for the dummy baseline.
    • Distance-only features barely separate the classes and fail especially on contradiction.
    • The separator gives only minor gains in this setting.

Interesting ideas

  • I1. Holding out an entire topic or event during cross-validation shows whether a model has learned the relation itself or just memorised topic-specific vocabulary.
  • I2. Reporting an ordinary split next to a held-out-domain split, plus a dummy baseline, makes the size of shortcut learning visible as a single gap.
  • I3. Contradictions that hinge on one number or place are invisible to similarity measures over whole texts. Detecting them requires comparing specific facts.
  • I4. The "unknown" class is built from pairs about the same event, so topical overlap is deliberately useless as a signal.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate trained components with whole fables or whole textbook units held out, so topic leakage cannot inflate scores.
  • I2 (learn): Add a held-out-domain score and a majority-class score as standing negative controls in every evaluation report.
  • I3 (world): Detect contradiction by aligning entities and checking attributes like counts, locations and dates in the text's world state, rather than measuring overall closeness.
  • I4 (answer): Build reader tests with distractor sentences on the same story but unrelated claims, checking that answers come from meaning rather than shared words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Octavia-Maria Şulea. 2017. Recognizing Textual Entailment in Twitter Using Word Embeddings. Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5306

@inproceedings{sulea2017recognizing,
  title = {{Recognizing Textual Entailment in Twitter Using Word Embeddings}},
  author = {Octavia-Maria Şulea},
  booktitle = {Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP},
  year = {2017},
  doi = {10.18653/v1/w17-5306},
  url = {https://doi.org/10.18653/v1/w17-5306}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015. JMLR.org, volume 37 of JMLR Workshop and Conference Proceedings, pages 957–966. http://jmlr.org/proceedings/papers/v37/kusnerb15.html. ↗
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Lluı́s Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. The Association for Computational Linguistics, pages 632–642. http://aclweb.org/anthology/D/D15/D151075.pdf. ↗
  3. Piroska Lendvai, Isabelle Augenstein, Kalina Bontcheva, and Thierry Declerck. 2016. Monolingual social media datasets for detecting contradiction and entailment. In Nicoletta Calzolari (Conference Chair), Khalid Choukri (Conference Chair), Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Hlne Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). ELRA, ELRA, 9, rue des Cordelires, 75013 Paris, 5/2016. find
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment. Springer-Verlag, Berlin, Heidelberg, MLCW’05, pages 177–190. find
  5. Joseph Lilleberg, Yun Zhu, and Yanqing Zhang. 2015. Support vector machines and word2vec for text classification with semantic features. In Ning Ge, Jianhua Lu, Yingxu Wang, Newton Howard, Philip Chen, Xiaoming Tao, Bo Zhang, and Lotfi A. Zadeh, editors, 14th IEEE International Conference on Cognitive Informatics & Cognitive Computing, ICCI*CC 2015, Beijing, China, July 6-8, 2015. IEEE Computer Society, pages 136–140. https://doi.org/10.1109/ICCI-CC.2015.7259377. ↗
  6. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Kathleen McKeown, Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui, editors, ACL 2008, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics, June 15-20, 2008, Columbus, Ohio, USA. The Association for Computer Linguistics, pages 1039–1047. http://www.aclweb.org/anthology/P08-1118. ↗
  7. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.. pages Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. CoRR abs/1612.03651. http://arxiv.org/abs/1612.03651. ↗
  8. Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In Francis R. for Computational Linguistics, pages 1442–1451. http://aclweb.org/anthology/N/N16/N16-1170.pdf.3111–3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality. ↗
  9. Will Y. Zou, Richard Socher, Daniel M. Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, pages 1393–1398. http://aclweb.org/anthology/D/D13/D13-1141.pdf. ↗
  10. Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Trans. Audio, Speech & Language Processing 24(4):694–707. https://doi.org/10.1109/TASLP.2016.2520371. ↗
  11. Ofir Pele and Michael Werman. 2008. A linear time histogram metric for improved sift matching. In Computer Vision–ECCV 2008. Springer, pages 495– 508. find
  12. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, pages 45–50. http://is.muni.cz/ publication/884893/en. ↗
  13. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. CoRR abs/1509.06664. http://arxiv.org/abs/1509.06664. ↗
  14. U Sauerland. 2007. Presupposition and Implicature in Compositional Semantics. ↗
  15. Rebecca Sharp, Mihai Surdeanu, Peter Jansen, Peter Clark, and Michael Hammond. 2016. Creating causal embeddings for question answering with minimal supervision. CoRR abs/1609.08097. http://arxiv.org/abs/1609.08097. ↗
  16. Octavia-Maria Sulea, Sergiu Nisioi, and Liviu P. Dinu. 2016. Using word embeddings to translate named entities. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016.. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2016/summaries/1167.html. ↗
  17. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688. http://arxiv.org/abs/1605.02688. ↗
  18. Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. The Association

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sulea-2017-recognizing-textual-entailment-twitter-word/