Elamur Research⌕
☰
home / papers / 2017 / Question Generation for Question Answering

Question Generation for Question Answering

Summary

Supervised QA systems need labeled question–answer pairs, and such annotation is costly. Duan et al. propose training a question generation (QG) model without manual labeling by mining Community-QA data from YahooAnswers. Groups of related questions are compared to find the shared n-gram (the "question topic"). Removing that topic leaves a reusable template with a "#" slot, called a question pattern. Pattern–answer–topic triples harvested from QA pairs then serve as training data for two QG variants:

  • a retrieval approach that ranks existing patterns with an attention-based CNN;
  • a generation approach that decodes patterns with an attentive BiGRU sequence-to-sequence model.

Both variants then fill the slot with a phrase from the passage and rank the candidate questions. The generative variant gets higher BLEU 4 scores on SQuAD, MS MARCO and WikiQA. The authors add similarity to generated questions as an extra feature in an answer sentence selection system (DocChat), which yields modest gains on all three benchmarks. The work shows that QG can strengthen QA using only freely available user data.

Key points

  • Mined data: 67,330,506 question clusters from YahooAnswers, which produced 20,818,569 question patterns. Of these, 105,623 patterns occur at least 10,000 times. A further 1,984,401 training instances were extracted from YahooAnswers QA pairs.

  • BLEU 4 with the crawled training data:

    ModelSQuADMS MARCOWikiQA
    G-QG (generation-based)12.3911.4613.57
    R-QG (retrieval-based)9.879.8611.38
    Seq2Seq-QG baseline (in-domain training sets)12.2810.462.04

  • Why G-QG wins: the two variants predict patterns about equally well. The difference lies in topic selection, which G-QG learns jointly through attention. R-QG relies on pre-trained topic embeddings, handles unseen topics poorly, and must score every pattern, which is expensive.

  • Error analysis: multi-fact errors account for 40% (the passage supports several valid questions but only one reference exists), paraphrase errors for 30%, and topic selection errors for 15%.

  • Effect on QA: adding the QG signal to DocChat raised MAP, MRR and ACC@1 on all three datasets, by about 0.4–1.2 points absolute. The authors note the gain was largest on MS MARCO, whose questions come from real Bing search queries.

Interesting ideas

  • I1. Using related-question clusters as weak supervision: an n-gram that recurs across a cluster of related questions marks the topic, and what remains is the template. Pattern and entity extraction emerge without any dictionary or knowledge base. This could be applied to Ukrainian forums or FAQ sites to build intent templates cheaply.
  • I2. Splitting questions into "pattern + slot": separating the question intent (template) from the entity (slot filler) keeps generation controllable. Restricting templates to frequent ones keeps them close to real user needs. The same design suits chatbots and search-query suggestion.
  • I3. Filtering harvested pairs by topic overlap: a crawled pair is kept only if the topic also appears in the answer. This one cheap consistency check removes many noisy pairs from web-scraped data.
  • I4. Generated questions as a reranking feature: for a candidate answer, generate questions and measure how similar they are to the user's query. As a sanity check, the authors built correct-answer and wrong-answer variants of the WikiQA test pairs. Questions generated from correct answers matched the labeled questions better (BLEU 0.2031 vs 0.1301). This test is reusable for validating any QG-for-QA feature.
  • I5. Evaluating QG with multiple references: the error analysis shows single-reference BLEU penalizes valid alternative questions. Datasets for QG evaluation should therefore include several questions and paraphrases per passage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1090

@inproceedings{duan2017question,
  title = {{Question Generation for Question Answering}},
  author = {Duan, Nan and Tang, Duyu and Chen, Peng and Zhou, Ming},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1090},
  url = {https://doi.org/10.18653/v1/d17-1090}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Linfeng Song and Lin Zhao. 2016. Domain-specific question generation from a knowledge base. In arXiv. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. Kateryna Tymoshenko, Daniele Bonadiman, and Alessandro Moschitt. 2016. Convolutional neural networks vs. convolution kernels: Feature engineering for answer sentence reranking. In NAACL. find
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP. find
  5. Jonathan Berant and Percy Liang. 2014. parsing via paraphrasing. In ACL. find
  6. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In ICLR. find our card
  7. Semantic Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. In arXiv. find
  8. Zhao Yan, Nan Duan, Junwei Bao, Peng Chen, Ming Zhou, Zhoujun Li, and Jianshe Zhou. 2016. Doc-chat: An information retrieval approach for chatbot engines using unstructured documents. In ACL. find
  9. Yllias Chali and Sadid Hasan. 2015. Towards topic-to-question generation. In CL. find
  10. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In EMNLP. find our card
  11. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Association for Computational Linguistics (ACL). find our card
  12. Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012. Semantics-based question generation and implementation. In Dialogue and Discourse. find
  13. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In NIPS. find
  14. Wenpeng Yin and Hinrich Schtze. 2017. Task-specific attentive pooling of phrase alignments contributes to sentence matching. In EACL. find
  15. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In ACL. find our card
  16. Wenpeng Yin, Hinrich Schtze, Bing Xiang, and Bowen Zhou. 2016. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. In TACL. find our card
  17. Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In ACL. find
  18. Lei Yu, Karl Morizt Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. In NIPS Workshop. find
  19. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In EMNLP. find
  20. Matthew Zeiler. 2012. Adadelta: an adaptive learning rate method. In CoRR abs. find our card
  21. Tri Nguyen, Mir Rosenberg, Xia Song, Jiangfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. In NIPS. find our card
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  23. Xipeng Qiu and Xuanjing Huang. 2015. Convolutional neural tensor network architecture for community based question answering. In IJCAI. find
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  25. Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016. Attentive pooling networks. In arXiv. find
  26. Iulian Vlad Serban, Alberto Garca-Durn, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questionswith recurrent neural networks: The 30m factoid question-answer corpus. In ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/duan-2017-question-generation-question-answering/