Elamur Research⌕
☰
home / papers / 2018 / Interpretation of Natural Language Rules in Conversational Machine Reading

Interpretation of Natural Language Rules in Conversational Machine Reading

Summary

The paper presents conversational machine reading (CMR), a task where answers are not stated literally in the text. Instead, a system must apply natural-language rules, such as those on UK government websites, to a user's situation. Given a question, a user scenario, a rule snippet, and the dialog history so far, the system either answers Yes, No, or Irrelevant, or asks a clarification question when key information is missing. To build data for this, the authors designed a crowdsourcing protocol. Annotators talk with a "virtual user" that gives random yes/no replies, which lets the authors explore different branches of the rule's decision tree. Other annotators then write scenarios based on parts of these dialogs. The resulting corpus, ShARC, has over 32k utterances. Baselines include end-to-end seq2seq networks (with and without copying) and a pipeline of separate classifiers, a rule-based question generator, and entailment components. The pipeline performed best. All models struggled with interpreting scenarios, which shows that reasoning with background knowledge is still largely unsolved.

Key points

  • The ShARC corpus contains over 32k CMR utterances, of which 21,890 are used for training. Its rule texts span topics including benefit programs, road traffic regulations, and descriptions of grants.
  • The dialog annotation is reliable, reaching a three-way Fleiss' Kappa of 0.71.
  • At each turn, the system outputs YES, NO, IRRELEVANT, or a free-form follow-up question.
  • The Combined Model beat the end-to-end neural model on CMR. It chains a Random Forest classifier, rule-based follow-up generation, and a Surface LR entailment model. Entailment baselines scored poorly on scenario interpretation, especially in macro accuracy.
  • In an oracle user study, people helped by a CMR agent answered more than twice as fast. They were also more accurate than people who read the rules themselves (93% vs 68%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1233

@inproceedings{saeidi2018interpretation,
  title = {{Interpretation of Natural Language Rules in Conversational Machine Reading}},
  author = {Saeidi, Marzieh and Bartolo, Max and Lewis, Patrick and Singh, Sameer and Rocktäschel, Tim and Sheldon, Mike and Bouchard, Guillaume and Riedel, Sebastian},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1233},
  url = {https://doi.org/10.18653/v1/d18-1233}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Antoine Bordes and Jason Weston. 2016. ing end-to-end goal-oriented dialog. abs/1605.07683. find
  3. Learn-CoRR, Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  4. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC : Question Answering in Context. In EMNLP. ArXiv: 1808.07036. find our card
  5. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2. find
  6. Sylvain Delisle, Ken Barker, Jean franois Delannoy, Stan Matwin, and Stan Szpakowicz. 1994. From text to horn clauses: Combining linguistic analysis and machine learning. In In 10th Canadian AI Conf, pages 9–16. find
  7. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. CoRR, abs/1603.06393. find our card
  8. Saeed Hassanpour, Martin O’Connor, and Amar Das. 2011. A framework for the automatic extraction of rules from online text. find
  9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701. find our card
  10. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based Neural Structured Learning for Sequential Question Answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1:1821–1831. find our card
  11. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa… find our card
  12. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115. find
  13. Andrew M. Olney, Arthur Graesser, and Natalie Person. 2012. Question generation from concept maps. 3. find
  14. Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. CoRR, abs/1603.06059. find our card
  15. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  16. Sudha Rao and Hal Daume III. 2018. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746, Melbourne, Austr… find
  17. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. CoQA: A Conversational Question Answering Challenge. arXiv:1808.07042 [cs]. ArXiv: 1808.07042 Citation Key: reddyCoQAConversation-alQuestion2018. find our card
  18. Vasile Rus, Paul Piwek, Svetlana Stoyanchev, Brendan Wyse, Mihai Lintean, and Cristian Moldovan. 2011. Question generation shared task and evaluation challenge: Status report. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 318–320, Stroudsburg, PA, USA.… find
  19. Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph. find
  20. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In The International Conference on Learning Representations (ICLR). find our card
  21. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2018. A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version. Dialogue & Discourse, 9(1):1–49. find
  22. Kenneth Silvestro. 1988. Using explanations for knowledge-base acquisition. International Journal of Man-Machine Studies, 29(2):159 – 169. find
  23. B. Moulin and D. Rousseau. 1992. Automated knowledge acquisition from regulatory texts. IEEE Expert, 7(5):27–35. find
  24. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing, pages 254–263. Association for Computational Lingui… find our card
  25. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933. find our card
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  27. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  28. Lucy Vanderwende. 2008. The importance of being important: Question generation. In In Proceedings of the Workshop on the Question Generation Shared Task and Evaluation Challenge. find
  29. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Fastqa: A simple and efficient neural architecture for question answering. CoRR, abs/1703.04816. find
  30. Joseph Weizenbaum. 1966. ELIZAa computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45. find
  31. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of ACL, abs/1710.06481. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saeidi-2018-interpretation-natural-language-rules-conversational/