Elamur Research⌕
☰
home / papers / 2019 / Unsupervised Question Answering by Cloze Translation

Unsupervised Question Answering by Cloze Translation

Paper · 2019 · Patrick Lewis, Ludovic Denoyer and Sebastian Riedel

Summary

Extractive QA systems usually depend on large, manually annotated question–answer collections, which exist for only a few languages and domains. Lewis, Denoyer and Riedel ask whether such a system can be trained with no aligned (question, context, answer) data. They build a synthetic data generator in three steps. First, they sample paragraphs from English Wikipedia. Next, they pick noun phrases or named entity mentions as answers and mask them to form cloze statements. Finally, they turn these clozes into questions. For that last step, they compare four options: identity mapping, noise injection, the Heilman and Smith rule-based generator, and an unsupervised seq2seq translator. The translator is trained on unpaired corpora of clozes and real questions using denoising auto-encoding and online back-translation. A standard QA model trained only on this generated data reaches 56.4 F1 on SQuAD v1, which beats early supervised baselines. The approach also helps when only a few labelled examples exist. This makes it relevant for under-resourced settings, though its best variant still relies on an NER system and a parser.

Key points

  • Without any SQuAD training data, the model scores 56.4 F1 on SQuAD v1, and 64.5 F1 on questions whose answer is a named entity.
  • In a few-shot setting with 32 labelled examples, the approach gives 59.3 F1, compared with 40.0 F1 without it.
  • The cloze corpus contains 5M items drawn from Wikipedia. The best pipeline uses NER trained on OntoNotes 5 and a constituency parser trained on the Penn Treebank.
  • The crude "noisy cloze" variant almost matches the best system, even though only 2.7% of its questions are well-formed. This suggests SQuAD can be solved largely through word overlap.
  • The rule-based translator performs worse than both the noise-based and unsupervised seq2seq approaches. Data and models are released at github.com/facebookresearch/UnsupervisedQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🎓University College LondonLondon · United Kingdom59 works · 16,463 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Question answering · 242 works

Cite

Cite as Patrick Lewis, Ludovic Denoyer and Sebastian Riedel. 2019. Unsupervised Question Answering by Cloze Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1484

@inproceedings{lewis2019unsupervised,
  title = {{Unsupervised Question Answering by Cloze Translation}},
  author = {Patrick Lewis, Ludovic Denoyer and Sebastian Riedel},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1484},
  url = {https://doi.org/10.18653/v1/p19-1484}
}

References

48 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Unsupervised statistical machine translation. In EMNLP, pages 3632–3642. Association for Computational Linguistics. our card
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics. our card
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022. our card
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching Word Vectors with Subword Information. arXiv:1607.04606 [cs]. ArXiv: 1607.04606. our card
  5. Bo Chen, Bo An, Le Sun, and Xianpei Han. 2018. Semi-Supervised Lexicon Learning for Wide-Coverage Semantic Parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 892–904, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  6. Christopher Clark and Matt Gardner. 2017. Simple and Effective Multi-Paragraph Reading Comprehension. arXiv:1710.10723 [cs]. ArXiv: 1710.10723. our card
  7. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 160–167, New York, NY, USA. ACM. find
  8. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. CoRR, abs/1710.04087. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. our card
  10. Bhuwan Dhingra, Danish Danish, and Dheeraj Rajagopal. 2018. Simple and Effective Semi-Supervised Question Answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 582–5… find
  11. Xinya Du and Claire Cardie. 2018. Harvesting Paragraph-level Question-Answer Pairs from Wikipedia. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1907–1917, Melbourne, Australia. Association for Computational Linguistics. our card
  12. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural Question Generation for Reading Comprehension. our card
  13. Manaal Faruqui and Dipanjan Das. 2018. Identifying Well-formed Natural Language Questions. arXiv:1808.09419 [cs]. ArXiv: 1808.09419. find
  14. Michael Heilman and Noah A. Smith. 2010. Good Question! Statistical Ranking for Question Generation. In Human Language Technologies: The 2010 find
  15. Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 609–617, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  16. Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In Proceedings of the 22Nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’99, pages 50–57, New York, NY, USA. ACM. find
  17. Tom Hosking and Sebastian Riedel. 2019. Evaluating Rewards for Question Generation Models. arXiv:1902.11049 [cs]. ArXiv: 1902.11049. find
  18. Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. our card
  19. Divyansh Kaushik and Zachary C. Lipton. 2018. How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks. arXiv:1808.04926 [cs, stat]. ArXiv: 1808.04926. our card
  20. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc… find
  21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… our card
  22. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv:1901.07291 [cs]. ArXiv: 1901.07291. find
  23. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised Machine Translation Using Monolingual Corpora Only. our card
  24. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-Based & Neural Unsupervised Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049, Brussels, Belgium. Association for Computat… find
  25. Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Kateryna Tymoshenko, Alessandro Moschitti, and Llus Mrquez. 2016. Semi-supervised Question Retrieval with Gated Convolutions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisti… find
  26. Mike Lewis and Angela Fan. 2019. Generative question answering: Learning to answer the whole question. In International Conference on Learning Representations. find
  27. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology -HLT ’94, page 114, Plainsboro, NJ. Associa… our card
  28. Todor Mihaylov and Anette Frank. 2018. Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 821–832, Melbourne, Australia. Assoc… our card
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa… our card
  30. Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and Robust Question Answering from Minimal Context over Documents. arXiv:1805.08092 [cs]. ArXiv: 1805.08092. our card
  31. Andrew M. Olney, Arthur C. Graesser, and Natalie K. Person. 2012. Question Generation from Concept Maps. Dialogue & Discourse, 3(2):75–99–99. find
  32. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In In EMNLP. our card
  33. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  34. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  35. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Dont Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Ling… our card
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. our card
  37. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. CoQA: A Conversational Question Answering Challenge. arXiv:1808.07042 [cs]. ArXiv: 1808.07042 Citation Key: reddyCoQAConversa-tionalQuestion2018. our card
  38. Vasile Rus, Brendan Wyse, Paul Piwek, Mihai Lintean, Svetlana Stoyanchev, and Cristian Moldovan. 2010. The First Question Generation Shared Task Evaluation Challenge. In Proceedings of the 6th International Natural Language Generation Conference, INLG ’10, pages 251–257, Stroudsburg, PA, USA. Associ… find
  39. Robyn Speer, Joshua Chin, and Catherine Havasi. 2016. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. arXiv:1612.03975 [cs]. ArXiv: 1612.03975. find
  40. Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A Minimal Span-Based Neural Constituency Parser. arXiv:1705.03919 [cs]. ArXiv: 1705.03919. our card
  41. Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y.-Lan Boureau. 2018. Multiple-Attribute Text Style Transfer. arXiv:1811.00552 [cs]. ArXiv: 1811.00552. find
  42. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What Makes Reading Comprehension Questions Easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208–4219, Brussels, Belgium. Association for Computational Linguistics. our card
  43. Liang Wang, Sujian Li, Wei Zhao, Kewei Shen, Meng Sun, Ruoyu Jia, and Jingming Liu. 2018. Multi-Perspective Context Aggregation for Semi-supervised Cloze-style Reading Comprehension. In Proceedings of the 27th International Conference on Computational Linguistics, pages 857–867, Santa Fe, New Mexico… find
  44. Dirk Weissenborn, Tom Koisk, and Chris Dyer. 2017. Dynamic Integration of Background Knowledge in Neural NLU Systems. ArXiv: 1706.02596. arXiv:1706.02596 [cs]. — [arXiv:1706.02596](https://arxiv.org/abs/1706.02596) find
  45. Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-Supervised QA with Generative Domain-Adaptive Nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1040–1050, Vancouver, Canada. Association for Comp… find
  46. Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012. Semantics-based Question Generation and Implementation. D&D, 3:11–42. find
  47. Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Sandeep Subramanian, Saizheng Zhang, and Adam Trischler. 2017. Machine Comprehension by Text-to-Text Neural Question Generation. arXiv:1705.02012 [cs]. ArXiv: 1705.02012. find
  48. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3901–3910, Brussels, Belgium. Association for C… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lewis-2019-unsupervised-question-answering-cloze-translation/