Elamur Research
☰
home / papers / 2020 / LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2007.08124

Summary

Reading comprehension benchmarks have grown harder by adding multi-hop evidence and commonsense knowledge, but deductive logical reasoning has received little attention in modern NLP. LogiQA fills this gap with 8,678 multiple-choice items, each pairing a passage and a question with four candidate answers. The items come from publicly available logic tests for civil servant examinations, written by domain experts rather than crowd workers. The authors manually excluded problems that rely on figures, charts or heavy mathematics, and kept a broad mix of reasoning types. Because answers usually do not appear verbatim in the passage and little outside knowledge is needed, success depends on inference rather than surface matching. Several neural readers were tested, from attention-based models to BERT and RoBERTa. The best system reached only 35.31% accuracy, against a human ceiling of 96.00%. This large gap makes LogiQA a demanding test for machine reading. It also offers a way to revisit classic logical AI questions with deep learning methods.

Key points

  • Dataset: 8,678 four-option questions from expert-authored civil servant logic exams, released at github.com/lgw863/LogiQA-dataset.
  • Reasoning types: Items can carry more than one label. Shares are categorical 30.8%, sufficient conditional 27.6%, necessary conditional 24.7%, conjunctive 21.3% and disjunctive 18.5%.
  • Baselines: The evaluated systems were BERT, RoBERTa, the Co-Matching Network, the Gated-Attention Reader, the Stanford Attentive Reader and a simple word-matching method. The best model scored 35.31%, close to the 25% chance level, while humans scored 96.00%.
  • RoBERTa by type: Categorical reasoning reached 55.00%. The other four types scored only about 17–23%. Errors included ignoring negation and failing to enumerate constraint combinations.
  • Lexical overlap analysis: The candidate with the highest word overlap is correct in only 28.37% of cases. Every model chose such candidates more often than that, showing a bias toward surface matching. RoBERTa showed the weakest bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

263 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 247 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2007.08124 · arXiv 2007.08124

@misc{liu2020logiqa,
  title = {{LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning}},
  author = {Liu, Jian and Cui, Leyang and Liu, Hanmeng and Huang, Dandan and Wang, Yile and Zhang, Yue},
  year = {2020},
  doi = {10.48550/arxiv.2007.08124},
  eprint = {2007.08124},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.08124}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Bramsen, M. Escobar-Molano, A. Patel, and R. Alonso. Extracting social power relationships from natural language. In NAACL, 2011. find
  2. Z. Cai, L. Tu, and K. Gimpel. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL, 2017. find
  3. D. Chen, J. Bolton, and C. D. Manning. A thorough examination of the CNN/daily mail reading comprehension task. In ACL, 2016. find our card
  4. A. Colmerauer and P. Roussel. The birth of prolog. In History of programming languages—II. ACM, 1996. find
  5. Y. Cui, W. Che, T. Liu, B. Qin, Z. Yang, S. Wang, and G. Hu. Pre-training with whole word masking for chinese bert. arXiv, 2019. find our card
  6. L. Cui, Y. Wu, S. Liu, Y. Zhang, and M. Zhou. Mutual: A dataset for multi-turn dialogue reasoning. In ACL, 2020. find
  7. J. Devlin, M-W Chang, K. Lee, and K. Toutanova. BERT: Pre-training of bidirectional transformers for language understanding. In NAACL, 2019. find
  8. B. Dhingra, H. Liu, Z. Yang, W. Cohen, and R. Salakhutdinov. Gated-attention readers for text comprehension. ACL, 2017. find
  9. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019. find our card
  10. I. Habernal, H. Wachsmuth, I. Gurevych, and B. Stein. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In NAACL, 2018. find
  11. K. Moritz Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In NIPS, 2015. find our card
  12. L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi. Cosmos QA: Machine comprehension with contextual commonsense reasoning. In EMNLP, 2019. find
  13. Patrick J. Hurley. A concise introduction to logic. Nelson Education, 2014. find
  14. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. find our card
  15. D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Etzioni, and D. Roth. Question answering via integer programming over semi-structured knowledge. arXiv, 2016. find
  16. Tomáš Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, Gábor Melis, and E. Grefenstette. The NarrativeQA reading comprehension challenge. TACL, 2018. find our card
  17. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. RACE: Large-scale Reading Comprehension dataset from Examinations. In EMNLP, 2017. find our card
  18. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv, 2019. find
  19. John McCarthy. Artificial intelligence, logic and formalizing common sense. In Philosophical logic and artificial intelligence. 1989. find
  20. Nils J. Nilsson. Logic and artificial intelligence. Artificial intelligence, 1991. find
  21. T. Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In ACL, 2019. find
  22. S. Ostermann, M. Roth, A. Modi, S. Thater, and M. Pinkal. Task 11: Machine comprehension using commonsense knowledge. In SemEval, 2018. find
  23. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. find our card
  24. M. Richardson, C.J.C. Burges, and E. Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, 2013. find our card
  25. A. Saha, R. Aralikatte, Mitesh M. Khapra, and K. Sankaranarayanan. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In ACL, 2018. find our card
  26. M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social iqa: Commonsense reasoning about social interactions. EMNLP, 2019. find
  27. K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. In EMNLP, 2019. find
  28. S. Wang, M. Yu, J. Jiang, and S. Chang. A co-matching model for multi-choice reading comprehension. In ACL, 2018. find
  29. J. Welbl, P. Stenetorp, and S. Riedel. Constructing datasets for multi-hop reading comprehension across documents. TACL, 2018. find our card
  30. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, Rémi Louf, M. Funtowicz, and J. Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, 2019. find our card
  31. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018. find our card
  32. Wen-tau Yih, Ming-Wei Chang, C. Meek, and A. Pastusiak. Question answering using enhanced lexical semantic models. In ACL, 2013. find