Elamur Research
☰
home / papers / 2020 / Break It Down: A Question Understanding Benchmark

Break It Down: A Question Understanding Benchmark

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.13.pdf

Summary

Complex questions in visual QA, reading comprehension, and database querying often have the same compositional structure. Until now, each task has handled question understanding separately. The authors propose question understanding as a standalone task. They introduce QDMR (Question Decomposition Meaning Representation), which writes a question as an ordered list of simple natural-language steps. Later steps can refer back to the results of earlier ones. QDMR has 13 operators inspired by SQL, and it assumes an idealized knowledge base, so it does not depend on any particular modality. Because the steps are plain language, crowd-workers without formal training can write them. Using a crowdsourcing pipeline, the team built the BREAK dataset of roughly 84K annotated questions. QDMR proved useful in two ways. It improved open-domain multi-hop QA on HotpotQA, and it can be mapped automatically to a pseudo-SQL language, which reduces the cost of annotating for semantic parsing. A CopyNet parser trained on BREAK beat several baselines but remains well below human quality. The work offers a shared, cross-task resource for studying multi-step reasoning.

Key points

  • BREAK contains 83,978 question–QDMR pairs drawn from 10 source datasets across three modalities (databases, images, text). Validation found 97.4% of the annotations correct.
  • QDMR defines 13 operator types, including Select, Filter, Project, Aggregate, Group, Superlative, Comparative, Union, Intersection, Discard, Sort, Boolean, and Arithmetic. A decomposition can be viewed as a DAG.
  • On HotpotQA, combining QDMR decompositions with a reading comprehension model (Min et al., 2019b) increased F1 from 43.3 to 52.4.
  • Among the parsing models (RuleBased, Seq2Seq, S2SDynamic, CopyNet), CopyNet performed best. On the test set it reached about 0.157 exact match and 0.746 SARI. In a manual check of its predictions, 54% were correct: 24% were exact matches and another 30% were semantically equivalent.
  • The dataset, models, and code are public at github.com/tomerwolgithub/Break.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00309

@article{wolfson2020break,
  title = {{Break It Down: A Question Understanding Benchmark}},
  author = {Wolfson, Tomer and Geva, Mor and Gupta, Ankit and Gardner, Matt and Goldberg, Yoav and Deutch, Daniel and Berant, Jonathan},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00309},
  url = {https://doi.org/10.1162/tacl_a_00309}
}

References

54 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In Association for Computational Linguistics (ACL). find our card
  2. Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2019. ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters. In North American Association for Computational Linguistics (NAACL). find
  3. In Association for Computational Linguistics (ACL). find
  4. Jianpeng Cheng, Siva Reddy, and Mirella Lapata. 2018. Building a neural semantic parser from a domain ontology. ArXiv, abs/1812.10037. Version 1. — [arXiv:1812.10037](https://arxiv.org/abs/1812.10037) find
  5. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL). find our card
  6. Eunsol Choi, Tom Kwiatkowski, and Luke Zettlemoyer. 2015. Scalable semantic parsing with partial ontologies. In Association for Computational Linguistics (ACL). find
  7. James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world’s response. In Computational Natural Language Learning (CoNLL), pages 18–27. find
  8. Ion Androutsopoulos, Graeme D. Ritchie, and Peter Thanisch. 1995. Natural language interfaces to databases – an introduction. Journal of Natural Language Engineering, 1:29–81. find
  9. Edgar F. Codd. 1970. A relational model of data for large shared data banks. Communications of the ACM, 13(6):377–387. find
  10. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), pages 2425–2433. find
  11. Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao, Alexander I. Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Workshop on Human Language Technology, pages 43–48. find
  12. Christopher Baik, Hosagrahar Visvesvaraya Jagadish, and Yunyao Li. 2019. Bridging the semantic gap with SQL query logs in natural language interfaces to databases. 2019 IEEE 35th International Conference on Data Engineering (ICDE), pages 374–385. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL). find
  14. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In 7th Linguistic Annotation Workshop and Interoperability with Discourse. find our card
  15. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL). find our card
  16. Donald D. Chamberlin and Raymond F. Boyce. 1974. SEQUEL: A structured English query language. In Proceedings of the 1974 ACM SIGFIDET (now SIGMOD) Workshop on Data Description, Access and Control, pages 249–264. ACM. find
  17. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke S. Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Association for Computational Linguistics (ACL). find our card
  18. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL). find our card
  19. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. In AllenNLP: A deep semantic natural language processing platform, arXiv, abs/1803.07640v2. find our card
  20. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. find
  21. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Association for Computational Linguistics (ACL). find our card
  22. Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Association for Computational Linguistics (ACL). find
  23. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Association for Computational Linguistics (ACL). find
  24. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR). find our card
  25. Nitish Gupta and Mike Lewis. 2018. Neural compositional denotational semantics for question answering. In Empirical Methods in Natural Language Processing (EMNLP). find
  26. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Empirical Methods in Natural Language Processing (EMNLP). find
  27. Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107. find
  28. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistic… find our card
  29. Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-loop parsing. In Empirical Methods in Natural Language Processing (EMNLP). find
  30. Fei Li and Hosagrahar Visvesvaraya Jagadish. 2014. NaLIR: An interactive natural language interface for querying relational databases. In International Conference on Management of Data, SIGMOD. find
  31. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In International Conference on Computer Vision (ICCV). find
  32. Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  33. Srini Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Association for Computational Linguistics (ACL). find
  34. Fei Li, Tianyin Pan, and Hosagrahar Visvesvaraya Jagadish. 2014. Schema-free SQL. In International Conference on Management of Data, SIGMOD, pages 1051–1062. find
  35. Percy Liang, Michael I. Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39:389–446. find
  36. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question–answer meaning representations. In North American Association for Computational Linguistics (NAACL). find our card
  37. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831. find our card
  38. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In Association for Computational Linguistics (ACL). find
  39. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In Association for Computational Linguistics (ACL). questions. In North American Association for Computational Linguistics (NAACL). find
  40. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302. find our card
  41. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics (ACL). find
  42. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. find our card
  43. Francis Jeffry Pelletier. 1994. The principle of semantic compositionality. Topoi, 13(1):11–24. find
  44. P. J. Price. 1990. Evaluation of spoken language systems: The ATIS domain. In Proceedings of the Third DARPA Speech and Natural Language Workshop, pages 91–95. find
  45. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan R. Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  46. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. Answering complex open-domain questions through iterative query generation. In Empirical Methods in Natural Language Processing (EMNLP), pages 2590–2602. Association for Computational Linguistics. find
  47. Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Association for Computational Linguistics (ACL). find
  48. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  49. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Empirical Methods in… find our card
  50. Siva Reddy, Oscar Täckström, Michael Collins, Tom Kwiatkowski, Dipanjan Das, Mark Steedman, and Mirella Lapata. 2016. Transforming dependency structures to logical forms for semantic parsing. In Association for Computational Linguistics (ACL). find
  51. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050–1055. find
  52. Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. Association for Computational Linguistics (ACL). find our card
  53. Luke Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI), pages 658–666. find
  54. Alon Talmor and Jonathan Berant. 2018. The web as knowledge-base for answering complex find