Elamur Research⌕
☰
home / papers / 2014 / The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

Paper · 2014 · Johannes Bjerva, Johan Bos, Rob van der Goot et al.

Summary

The paper describes the Groningen/Bologna entry to SemEval-2014 Shared Task 1. The task asked systems to label sentence pairs as entailment, contradiction or neutral, and to score how related the pairs are on a 1–5 scale. For entailment, the authors reused a logic-based pipeline. The C&C tools and Boxer build Discourse Representation Structures, which are then converted to first-order logic. The theorem prover Vampire looks for proofs and contradictions, and the model builders Paradox and Mace-2 run when no proof is found. WordNet hypernyms supply background axioms. PPDB paraphrases rewrite the first sentence when an initial proof attempt fails. This symbolic core is precise but misses many entailments, so a small classifier trained on relatedness scores re-examines the pairs labelled neutral. The final system reached about 82% accuracy and placed in the top five. For relatedness, a Random Forest combines features drawn mostly from the logical machinery. It achieved the lowest MSE of all participants. The work suggests that deep semantic representations can serve as useful signals for graded similarity, not only for yes/no inference.

Key points

  • Entailment accuracy by configuration:
    • most-frequent-class baseline: 56.7
    • logic core alone: 77.6
    • with PPDB paraphrasing: 79.6
    • with PPDB and neutral reclassification (scikit-learn SVM, one relatedness feature): 81.6
  • Inference setup:
    • Vampire gets a 30-second limit.
    • If it fails, Paradox fixes a domain size and Mace-2 then builds a minimal model of that size.
    • WordNet was involved in about 20% of training-set entailment proofs but only about 4% of contradiction proofs.
  • Paraphrase source: PPDB-1.0 XL gave the best precision/recall trade-off. Larger versions hurt precision, and smaller versions hurt recall.
  • Relatedness model:
    • 32 features fed to a Random Forest with 1000 trees, maximum depth 20 and at most 3 features per split.
    • Trained on 5000 pairs and tested on 4927.
    • The primary run scored Pearson 0.827 and MSE 0.322.
    • Logical-model features were the strongest single group (Pearson 0.737, MSE 0.476 with PPDB).
  • Other features:
    • DRS agent/patient overlap, noun/verb lemma overlap, WordNet novelty and synset overlap/distance.
    • The RTE label itself.
    • A word2vec skip-gram additive sentence vector with 1600 dimensions.
    • Sentence IDs, which leaked useful signal but were excluded from the primary run.

Interesting ideas

  • I1. Hypothesis-guided paraphrasing: paraphrases rewrite the premise only after a failed proof, and only when the source phrase is short, its target words are absent from the premise, and all target words occur in the hypothesis. This keeps paraphrasing narrowly targeted instead of noisy.
  • I2. Models as similarity evidence: when no proof exists, a minimal model of each sentence and of both together is built, and the overlap of their entities and relations measures relatedness. A failed inference still yields a graded, inspectable signal.
  • I3. Precise core plus a narrow recall booster: the logical system is trusted on its entailment and contradiction verdicts, and a statistical classifier only reconsiders pairs it called neutral.
  • I4. Keep the better of original and paraphrased features: each overlap feature is computed for both the original and the paraphrased pair, and the higher value is kept.
  • I5. Leakage found and set aside: the authors noticed that sentence IDs predicted relatedness and reported this separately rather than using it in the primary run.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): when a reader cannot match a question to the text world, it could retry with constrained rewrites that pull the text toward the question's own wording, and log each rewrite as an explicit step.
  • I2 (world): compare the worlds built from two texts by counting shared entities and relations, giving an explainable closeness score instead of vector cosine.
  • I3 (answer): let a learned scorer act only on "unknown" outcomes of the symbolic reader, so confident derivations are never overridden.
  • I4 (frames): when matching who-did-what-to-whom frames across sentences, score both the literal and the normalized variants (passive/active, negation substitutions) and keep the best alignment.
  • I5 (learn): add ordering and ID shortcuts as deliberate negative-control features in evaluation, so that any gain they produce flags a dataset artifact.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands49 works · 4,843 citations🎓University of BolognaBologna · Italy10 works · 1,708 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Johannes Bjerva, Johan Bos, Rob van der Goot and Malvina Nissim. 2014. The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/s14-2114

@inproceedings{bjerva2014meaning,
  title = {{The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity}},
  author = {Johannes Bjerva, Johan Bos, Rob {van der Goot} and Malvina Nissim},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2114},
  url = {https://doi.org/10.3115/v1/s14-2114}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Johan Bos and Katja Markert. 2006. Recognising textual entailment with robust logical inference. In Joaquin Quinonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc, editors, Machine Learning Challenges, MLCW 2005, volume 3944 of LNAI, pages 404–426. find
  2. Johan Bos. 2004. Computational Semantics in Discourse: Underspecification, Resolution, and Inference. Journal of Logic, Language and Information, 13(2):139–157. ↗
  3. Johan Bos. 2008. Wide-Coverage Semantic Analysis with Boxer. In J. Bos and R. Delmonte, editors, Semantics in Text Processing. STEP 2008 Conference Proceedings, volume 1 of Research in Computational Semantics, pages 277–286. College Publications. find
  4. Johan Bos. 2013. Is there a place for logic in recognizing textual entailment? Linguistic Issues in Language Technology, 9(3):1–18. ↗
  5. Wauter Bosma and Chris Callison-Burch. 2006. Paraphrase substitution for recognizing textual entailment. In Proceedings of CLEF. find
  6. Leo Breiman. 2001. Random forests. Machine learning, 45(1):5–32. find
  7. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27. find
  8. K. Claessen and N. Sörensson. 2003. New techniques that improve mace-style model finding. In P. Baumgartner and C. Fermüller, editors, Model Computation – Principles, Algorithms, Applications (Cade-19 Workshop), pages 11–27, Miami, Florida, USA. find
  9. James Curran, Stephen Clark, and Johan Bos. 2007. Linguistically Motivated Large-Scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 33–36, Prague, Czech Republic.
  10. Christiane Fellbaum, editor. 1998. WordNet. An Electronic Lexical Database. The MIT Press. find
  11. Juri Ganitkevitch, Benjamin VanDurme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2013), Atlanta, Georgia, June. Association for Computational Linguistics.
  12. Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic; An Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and DRT. Kluwer, Dordrecht. find
  13. M. Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini, and R. Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation.
  14. W. McCune. 1998. Automatic Proofs and Counterexamples for Some Ortholattice Identities. Information Processing Letters, 65(6):285–291. ↗
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  17. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of english. Journal of Natural Language Engineering, 7(3):207– 223. ↗
  18. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  19. A. Riazanov and A. Voronkov. 2002. The Design and Implementation of Vampire. AI Communications, 15(2–3):91–110. find
  20. Mark Steedman. 2001. The Syntactic Process. The MIT Press. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bjerva-2014-meaning-factory-formal-semantics-recognizing/