Elamur Research⌕
☰
home / papers / 2019 / Can You Unpack That? Learning to Rewrite Questions-in-Context

Can You Unpack That? Learning to Rewrite Questions-in-Context

Summary

In conversational question answering, later questions depend on earlier turns. Pronouns, dropped entities, and phrases like "anything else" only make sense given the dialogue history. The authors define a task called question-in-context rewriting: converting such a dependent question into a standalone version with the same answer. To support it, they built CANARD by crowdsourcing rewrites of QuAC questions on Mechanical Turk. Workers were told to make minimal edits, live JavaScript checks warned about leftover pronouns or vague words, and the authors manually reviewed flagged submissions. They compare three baselines using BLEU against human rewrites: an unchanged copy, a rule replacing the first pronoun with the article title, and an attention-based Seq2Seq LSTM with a copy mechanism. The neural model beats the simple baselines but stays roughly 9 BLEU below human agreement. Its typical errors are overly short rewrites, confusion between related entities, and dropped list items. The resource matters because it turns conversational QA into ordinary reading comprehension and supports future systems that ask users for clarification.

Key points

  • CANARD contains 40,527 rewritten questions derived from QuAC: 31,538 for training, 3,418 for development (from QuAC train), and 5,571 for testing (QuAC's full dev set).
  • Quality control flagged about 15% of the data for review, and 19.9% of submissions were rejected. In a manual check of 50 rewrites, every desired property appeared in at least 90% of cases.
  • BLEU scores (dev/test):
    • Copy: 33.84/36.25
    • Pronoun Sub: 47.72/47.44
    • Seq2Seq (OpenNMT, GloVE embeddings): 51.37/49.67
    • Human: 59.92, measured on a small validation subset
  • Pronouns appear in 53.9% of QuAC questions and in 75.0% of conversation histories. Mixed-category pronouns in the history affect 27.8% of the data, and only about a third of rewrites are reachable by pronoun replacement alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1605

@inproceedings{elgohary2019can,
  title = {{Can You Unpack That? Learning to Rewrite Questions-in-Context}},
  author = {Elgohary, Ahmed and Peskov, Denis and Boyd-Graber, Jordan},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1605},
  url = {https://doi.org/10.18653/v1/d19-1605}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  2. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2018. Ask the right questions: Active question reformulation with reinforcement learning. In Proceedings of the International Conference on Learning Representations. find
  3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  4. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Computer Vision and Pattern Recognition. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  6. Ye Liu, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2019. Generative question refinement with deep reinforcement learning in retrieval-based QA system. In Proceedings of the ACM International Conference on Information and Knowledge Management. find
  7. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing. find
  8. Denis Peskov, Joe Barrow, Pedro Rodriguez, Graham Neubig, and Jordan Boyd-Graber. 2019. Mitigating noisy inputs for question answering. In Proceedings of the Annual Conference of the International Speech Communication Association. find
  9. Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a. BERT with history modeling for conversational question answering. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. find
  10. Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft, and Mohit Iyyer. 2019b. Attentive history selection for conversational question answering. In Proceedings of the ACM International Conference on Information and Knowledge Management. find
  11. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  13. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  14. Pushpendre Rastogi, Arpit Gupta, Tongfei Chen, and Lambert Mathias. 2019. Scaling multi-domain dialogue state tracking via query reformulation. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  15. Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. 2018. Dataset and baselines for sequential open-domain question answering. In Proceedings of Empirical Methods in Natural Language Processing. find
  16. Shi Feng and Jordan Boyd-Graber. 2019. What AI can do for me: Evaluating machine learning interpretations in cooperative play. In International Conference on Intelligent User Interfaces. find
  17. Hsin-Yuan Huang, Eunsol Choi, and Wen tau Yih. 2019. FlowQA: Grasping flow in history for conversational machine comprehension. In Proceedings of the International Conference on Learning Representations. find
  18. Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander Rush. 2018. OpenNMT: Neural machine translation toolkit. In Proceedings of Association for Machine Translation in the Americas. find
  19. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics. find our card
  20. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Association for Computational Linguistics. find
  21. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of… find our card
  22. Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance ReWriter. In Proceedings of the Association for Computational Linguistics. find
  23. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of Empirical Methods in Natural Language Processing. find
  24. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics. find
  25. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  26. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. In Transactions of the Association for Computational Linguistics. find
  27. Bonnie Webber. 1992. Question answering. In Stuart C. Shapiro, editor, Encyclopedia of Artificial Intelligence, 2nd edition, pages 814–822. John Wiley & Sons, Inc., New York, NY. find
  28. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elgohary-2019-can-you-unpack-that-learning/