Elamur Research⌕
☰
home / papers / 2020 / Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

Paper · 2020 · Akari Asai and Hannaneh Hajishirzi

Summary

Comparison questions ask how two entities or events relate, for example which is larger or what effect a cause has. Neural QA models such as RoBERTa often answer them inconsistently. A model may give the same answer to a question and its antonym-flipped version, or break a chain of cause-effect reasoning. Such datasets are also costly to annotate, so they are small.

Asai and Hajishirzi address this with two logic-driven components. The first is data augmentation that creates new labeled examples using symmetry and transitivity rules. Symmetric examples come from antonym substitution or from adding or removing negation. Transitive examples come from chaining linked cause-effect questions. The second is a consistency regularizer added to the task loss, which penalizes predictions that violate these rules.

The method does not depend on any particular model. It was tested on three formats: WIQA (classification), QuaRel (multiple choice) and HotpotQA (span extraction). It raised accuracy, set new state-of-the-art results on WIQA and QuaRel, and sharply reduced inconsistent predictions. It also worked well with little labeled data.

Key points

  • Applied to RoBERTa, the method improves results by 1.0 points on WIQA, 5.0 on QuaRel and 2.5 on HotpotQA. It surpasses the previous state of the art on WIQA by 4.7 points and on QuaRel by 8.4.
  • Consistency violations fall by 58% on HotpotQA. On WIQA, symmetric violations drop from 13.9% to 8.3% and transitive violations from 10.0% to 2.5%.
  • With only 20% of the training data, gains over RoBERTa exceed 12 points on WIQA and 14 points on QuaRel. This matches a strong baseline trained on the full data.
  • In the ablation, a conventional WordNet-based paraphrasing augmentation without logic rules gave no notable gains. Combining logic-guided augmentation with regularization worked best.
  • Augmented samples (24,715 for WIQA, 836 for QuaRel and 3,538 for HotpotQA) are drawn once before training. Code and data are available at github.com/AkariAsai/logic_guided_qa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 243 works

Cite

Cite as Akari Asai and Hannaneh Hajishirzi. 2020. Logic-Guided Data Augmentation and Regularization for Consistent Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.499

@inproceedings{asai2020logic,
  title = {{Logic-Guided Data Augmentation and Regularization for Consistent Question Answering}},
  author = {Akari Asai and Hannaneh Hajishirzi},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.499},
  url = {https://doi.org/10.18653/v1/2020.acl-main.499}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. 2019. AutoAugment: Learning augmentation strategies from data. In CVPR. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  3. Ming Ding, Chang Zhou, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In ACL. find
  4. Jay Yoon Lee, Sanket Vaibhav Mehta, Michael Wick, Jean-Baptiste Tristan, and Jaime Carbonell. 2019. Gradient-based inference for networks with output constraints. In AAAI. find
  5. Tao Li, Vivek Cupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In EMNLP. find
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. find
  7. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM.
  8. Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. 2019. Be consistent! improving procedural text comprehension using label consistency. In NAACL. find
  9. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In ACL.
  10. Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In EMNLP. find
  11. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In ACL.
  12. Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11(Jul):2001–2049. find
  13. Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural NLI models to integrate logical background knowledge. In ACL. find
  14. Regina Paxton Gazes, Nicholas W Chee, and Robert R Hampton. 2012. Cognitive mechanisms for transitive inference performance in rhesus monkeys: Measuring the influence of associative strength and inferred order. Journal of Experimental Psychology: Animal Behavior Processes, 38(4):331. find
  15. Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. VQA-LOL: Visual question answering under the lens of logic. arXiv:2002.08325. — [arXiv:2002.08325](https://arxiv.org/abs/2002.08325) find
  16. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias for natural language inference by fitting the residual. In EMNLP Workshop on DeepLo. find
  17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
  18. Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. 2018. AdvEntuRe: Adversarial training for textual entailment with knowledge-guided examples. In ACL. find
  19. Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. 1983. Optimization by simulated annealing. science. find
  20. Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. Technical report, Citeseer. find
  21. Arindam Mitra, Chitta Baral, Aurgho Bhattacharjee, and Ishan Shrivastava. 2019. A generate-validate approach to answering questions about qualitative relationships. arXiv:1908.03645. — [arXiv:1908.03645](https://arxiv.org/abs/1908.03645) find
  22. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv:1904.01038.
  23. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv:1904.08779.
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
  25. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In EMNLP.
  26. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. QuaRel: A dataset and models for answering questions about qualitative relationships. In AAAI. find
  27. Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. 2019. WIQA: A dataset for ”what if...” reasoning over procedural text. In EMNLP. find
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv:1910.03771.
  29. Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. 2019. Adversarial examples improve image recognition. arXiv:1911.09665. — [arXiv:1911.09665](https://arxiv.org/abs/1911.09665) find
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
  31. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In ICLR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asai-2020-logic-guided-data-augmentation-regularization/