Elamur Research
☰
home / papers / 2017 / Search-based Neural Structured Learning for Sequential Question Answering

Search-based Neural Structured Learning for Sequential Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1167.pdf

Summary

Most semantic parsing benchmarks for table QA target long, heavily compositional questions that real users rarely ask. This paper argues that a complex information need is more naturally pursued as a short dialogue of simple, linked questions, and it builds resources for that setting. The authors crowdsourced SequentialQA (SQA). Workers broke complicated WikiTableQuestions items into chains of easier questions whose answers are table cells. The result is 6,066 sequences with 17,553 question–answer pairs over Wikipedia tables. To answer these chains, the authors introduce DynSP, a neural semantic parser that treats parsing as a state–action search and builds SQL-like queries step by step. Separate neural modules score each action type, and the network structure is assembled dynamically during search. Training uses a margin-based, reward-guided objective that works with weak supervision from answers alone. A dedicated "subsequent" statement lets a parse add constraints to the previous answer. On SQA, DynSP outperforms state-of-the-art systems built for complex questions, in both per-question and full-sequence accuracy. This shows the value of modeling conversational context.

Key points

  • SQA was built from 2,022 WikiTableQuestions (WTQ) intents of nine or more words. Three workers decomposed each intent, giving about 2.9 questions per sequence. The data uses an 83/17 train/test split, and no test tables appear in training. The dataset is available at aka.ms/sqa.
  • The paper identifies three main question types:
    • column selection (23%)
    • subset selection (27%)
    • row selection (19%)
    • The remaining 31% mix these types. Column selection drops from 51% of first questions to 18% of second questions.
  • Parses consist of a select statement plus conjoined conditions using these operators: =, ≠, >, ≥, <, ≤, arg min, arg max. A "subsequent" preamble handles follow-up questions.
  • DynSP draws on STAGG (search-based parsing) and dynamic neural module networks (DNMN). Unlike DNMN, its per-example network structure is not fixed in advance. It is compared against FP and the neural programmer (NP).
  • The error analysis highlights two main failure sources:
    • semantic matching errors, such as "not listed" versus "Unknown" in a table
    • questions that need several previous answers, which the parse language cannot express

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1167

@inproceedings{iyyer2017search,
  title = {{Search-based Neural Structured Learning for Sequential Question Answering}},
  author = {Iyyer, Mohit and Yih, Wen-tau and Chang, Ming-Wei},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1167},
  url = {https://doi.org/10.18653/v1/p17-1167}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  2. Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of Empirical Methods in Natural Language Processing. find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of Empirical Methods in Natural Language Processing. find
  4. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pages 1156– 1165. find
  5. Stefan L Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in Cognitive Science 5(3). find
  6. John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive Science 30(4). find
  7. Sanda Harabagiu, Andrew Hickl, John Lehmann, and Dan Moldovan. 2005. Experiments with interactive question-answering. In Proceedings of the Association for Computational Linguistics. find
  8. Diane Kelly and Jimmy Lin. 2007. Overview of the trec 2006 ciqa task. In ACM SIGIR Forum. ACM, volume 41, pages 107–116. find
  9. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition 106(3). find
  10. Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Van… find
  11. We thank the anonymous reviewers for their insightful comments. We are also grateful to Panupong Pasupat for his help in configuring the floating parser baseline, and to Arvind Neelakantan for his help Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms vi… find
  12. Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. 2017. Deep learning with dynamic computation graphs. In Proceedings of the International Conference on Learning Representations. find
  13. Arvind Neelakantan, Quoc Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In Proceedings of the International Conference on Learning Representations. find
  14. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau… find our card
  15. Luke Zettlemoyer and Michael Collins. 2009. Learning context-dependent mappings from sentences to logical form. In Proceedings of the Association for Computational Linguistics. find