Elamur Research⌕
☰
home / papers / 2019 / A Logic-Driven Framework for Consistency of Neural Models

A Logic-Driven Framework for Consistency of Neural Models

Paper · 2019 · Tao Li, Vivek Gupta, Maitrey Mehta et al.

Summary

Neural classifiers can score well on individual examples yet still hold contradictory beliefs across related inputs. The authors study this in natural language inference (NLI). They treat inconsistency as a broader form of prediction error and measure it against invariants written in first-order logic. Labeled examples are encoded as trivially true rules, so the two proposed metrics, global violation (ρ) and conditional violation (τ), reduce to ordinary error when only annotations are available.

To lower inconsistency during training, they relax the logic with t-norms, mainly the product t-norm, which turns the rules into differentiable losses. Under this mapping, standard cross-entropy appears as a special case. Symmetry and transitivity rules can then be enforced on unlabeled sentence pairs and triples without changing the model architecture.

Experiments use BERT trained on SNLI and MultiNLI. Adding the constraints sharply cuts violations while keeping test accuracy intact. This suggests that leaderboard accuracy alone is a poor measure of reasoning quality.

Key points

  • A BERT-based NLI model with roughly 90% F-score on SNLI still broke the "entails + contradicts ⇒ contradicts" rule in about 46% of unlabeled triples.
  • Three consistency types are encoded in logic:
    • annotation consistency, which yields cross-entropy;
    • symmetry of Contradiction, with a loss based on the absolute gap between log c(P,H) and log c(H,P);
    • four transitivity rules over sentence triples.
  • Product, Gödel and Łukasiewicz t-norms are compared conceptually, and the product t-norm is adopted.
  • On 100k evaluation triples, the constrained model reduced global transitivity violation from 0.7–1.8% to at most 0.2%. Conditional violation for N∧E→¬C fell from 9.0% to 1.8%.
  • Constraints push predictions on unlabeled data toward Neutral (about 60k → 84k of 100k pairs). Code is available at github.com/utahnlp/consistency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of UtahSalt Lake City · United States40 works · 6,803 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Tao Li, Vivek Gupta, Maitrey Mehta and Vivek Srikumar. 2019. A Logic-Driven Framework for Consistency of Neural Models. arXiv:1909.00126.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.00126 · arXiv 1909.00126

@misc{li2019logic,
  title = {{A Logic-Driven Framework for Consistency of Neural Models}},
  author = {Tao Li, Vivek Gupta, Maitrey Mehta and Vivek Srikumar},
  year = {2019},
  doi = {10.48550/arxiv.1909.00126},
  eprint = {1909.00126},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.00126}
}

References

40 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA Corpora Generation with Roundtrip Consistency.
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  3. Ming-Wei Chang, Lev Ratinov, and Dan Roth. 2012. Structured Learning with Constrained Conditional Models. Machine learning, 88(3):399–431. find
  4. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long Short-Term Memory-Networks for Machine Reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  5. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  7. Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen tau Yih, Peter Clark, and Claire Cardie. 2019. Be Consistent! Improving Procedural Text Comprehension using Label Consistency. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis… find
  8. Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. 2019. DL2: Training and Querying Neural Networks with Logic. In International Conference on Learning Representations. find
  9. Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior Regularization for Structured Latent Variable Models. Journal of Machine Learning Research. find
  10. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
  11. Madan M Gupta and J Qi. 1991. Theory of T-norms and fuzzy inference methods. Fuzzy Sets and Systems, 40. find
  12. Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
  13. Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing Deep Neural Networks with Logic Rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  14. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  15. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  16. Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. 2018. AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find
  17. Martin Kay. 1997. The Proper Place of Men and Machines in Language Translation. Machine Translation, 12. find
  18. Angelika Kimmig, Stephen Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2012. A short Introduction to Probabilistic Soft Logic. In Proceedings of the NIPS Workshop on Probabilistic Programming: Foundations and Applications. find
  19. Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations.
  20. Erich Peter Klement, Radko Mesiar, and Endre Pap. 2013. Triangular Norms. Springer Science & Business Media. find
  21. Tao Li and Vivek Srikumar. 2019. Augmenting Neural Networks with First-order Logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  22. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In 13th European Conference on Computer Vision. Springer.
  23. Nelson F Liu, Roy Schwartz, and Noah A Smith. 2019. Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  24. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing Revisited with Neural Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.
  25. Sanket Vaibhav Mehta, Jay Yoon Lee, and Jaime Carbonell. 2018. Towards Semi-Supervised Learning for Deep Semantic Role Labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  26. Pasquale Minervini and Sebastian Riedel. 2018. Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning. find
  27. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics. find
  28. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2018. Analyzing Compositionality-Sensitivity of NLI Models. The Thirty-Second AAAI Conference on Artificial Intelligence.
  29. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  30. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.
  31. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  33. Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. 2015. Injecting Logical Background Knowledge into Embeddings for Relation Extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  34. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional Attention Flow for Machine Comprehension. International Conference on Learning Representations.
  35. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research.
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems.
  37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
  38. Haohan Wang, Da Sun, and Eric P Xing. 2019. What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks. The Thirty-Third AAAI Conference on Artificial Intelligence. find
  39. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  40. Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. 2018. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In International Conference on Machine Learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2019-logic-driven-framework-consistency-neural/