Elamur Research⌕
☰
home / papers / 2020 / Semantic Graphs for Generating Deep Questions

Semantic Graphs for Generating Deep Questions

Paper · 2020 · Liangming Pan, Yuxi Xie, Yansong Feng et al.

Summary

Most question generation (QG) systems produce shallow, factoid questions that can be answered from one sentence. This paper defines Deep Question Generation (DQG), where a good question requires combining several separate pieces of evidence in a document, such as comparing facts from two Wikipedia paragraphs. The authors first convert the input into a semantic graph. Its nodes come from either semantic role labelling (SRL) or dependency parsing (DP), and typed relations link them within and across sentences. The graph is encoded with Att-GGNN, a Gated Graph Neural Network extended with attention so it can weigh different relation types. The resulting node representations are merged with word-level document encodings. The question decoder is trained jointly with an auxiliary content selection task that identifies question-worthy nodes. On HotpotQA, the model reaches state-of-the-art results on automatic metrics and in human evaluation, with the strongest gains on questions involving multiple facts. Error analysis shows that the graph substantially reduces semantically incoherent questions. The work is an early step toward QG that demands real reasoning, with uses in QA data augmentation, tutoring and dialogue systems.

Key points

  • Task and motivation: The paper presents DQG as a new task. The authors argue that SQuAD is unsuitable for evaluating it because over 80% of SQuAD questions depend on a single sentence.
  • Architecture: The model has four parts:
    • a document encoder;
    • an Att-GGNN semantic graph encoder, using SRL- or DP-based graphs;
    • a node-level content selector;
    • a question decoder with a copy mechanism, trained jointly with the selector.
  • Error reduction: The semantic error rate falls to 8.3% for the proposed model, compared with 26.4% for the CGC-QG baseline. Answerability, however, stays roughly on par with S2sa-at-mp-gsa.
  • Effect of content selection: Joint training raises the ratio of attention on relevant versus irrelevant graph nodes from 1.067 to 1.214. Attention entropy changes from 3.51 to 3.57.
  • Code availability: The code is published at github.com/WING-NUS/SG-Deep-Question-Generation.
  • Future work: The authors suggest richer semantic representations, such as discourse structure and knowledge graphs, and explicit modelling of reasoning chains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China100 works · 35,618 citations🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua and Min-Yen Kan. 2020. Semantic Graphs for Generating Deep Questions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.135

@inproceedings{pan2020semantic,
  title = {{Semantic Graphs for Generating Deep Questions}},
  author = {Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua and Min-Yen Kan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.135},
  url = {https://doi.org/10.18653/v1/2020.acl-main.135}
}

References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
  2. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 2306–2317.
  3. Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge text and knowledge by learning multi-prototype entity mention embedding. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1623–1633. find
  4. Yllias Chali and Sadid A. Hasan. 2012. Towards automatic topical question generation. In International Conference on Computational Linguistics (COLING), pages 475–492. find
  5. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP)… find
  6. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations (ICLR).
  7. Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1907– 1917.
  8. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1342–1352.
  9. Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 866–874.
  10. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Annual Meeting of the Association for Computational Linguistics (ACL).
  11. Michael Heilman. 2011. Automatic factual question generation from text. Language Technologies Institute School of Computer Science Carnegie Mellon University, 195. find
  12. Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4473– 4483. find
  13. Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In AAAI Conference on Artificial Intelligence (AAAI), pages 6602–6609. find
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). find
  15. Alon Lavie and Abhaya Agarwal. 2007. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT@ACL), pages 228–231.
  16. Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR). find
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
  18. Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243–3253. find
  19. Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. 2019a. Learning to generate questions by learning what not to generate. In International World Wide Web Conference (WWW), pages 1106–1118. find
  20. Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019b. Discourse representation parsing for sentences and documents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6248–6262. find
  21. Lluı́s Màrquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: An introduction to the special issue. Computational Linguistics, 34(2):145–159. find
  22. Lluı́s Màrquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: An introduction to the special issue. Computational Linguistics, 34(2):145–159. find
  23. Rik van Noord, Lasha Abzianidze, Antonio Toral, and Johan Bos. 2018. Exploring neural methods for parsing discourse representation structures. Transactions of the Association for Computational Linguistics (TACL), 6:619–633. find
  24. Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. CoRR, abs/1905.08949. — [arXiv:1905.08949](https://arxiv.org/abs/1905.08949) find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318.
  26. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  27. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4932– 4942.
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392.
  29. Xingdi Yuan, Tong Wang, Çaglar Gülçehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In The 2nd Workshop on Representation Learning for NLP (Rep4NLP@ACL), pages 15–25. find
  30. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3901– 3910.
  31. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In CCF International Conference of Natural Language Processing and Chinese Computing (NLPCC), pages 662–671. find
  32. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073–1083. find
  33. Peng Shi and Jimmy Lin. 2019. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1904.05255.
  34. Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3930–3939.
  35. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL).
  36. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph attention networks. CoRR, abs/1710.10903.
  37. Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10685– 10694. find
  38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2369–2380.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2020-semantic-graphs-generating-deep-questions/