Elamur Research⌕
☰
home / papers / 2019 / MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

Summary

Many extractive reading comprehension (RC) benchmarks now exist, but it was unclear whether a model trained on one transfers to another, or whether existing data can reduce the work of building a new dataset. Talmor and Berant study this across 10 RC datasets, all converted to a single-turn extractive format. They compare DocQA, which uses GloVe inputs, with a BERT-based reader called BertQA. They test generalization, where no target examples are used, and transfer, where the model is fine-tuned on target data.

Models tend to overfit their training set. BERT improves generalization, mostly on Wikipedia and news text and much less on web snippets. Performance drops when source and target differ in document language, question style or the reasoning phenomenon they probe, and the losses grow as these mismatches combine. Pre-training on other RC data still helps even with BERT. Mixing several sources matches target-only training with about three times fewer target examples. The resulting model, MultiQA (BERT-large trained on multiple datasets), reaches state of the art on five datasets. For teams building new QA resources with limited annotation budgets, this is practically useful.

Key points

  • The study covers five large datasets (SQuAD, NewsQA, SearchQA, TriviaQA, HotpotQA), each capped at 75K training examples to control for size. It also uses five small ones (CQ, CWQ, ComQA, WikiHop, DROP), and only DROP's 33,573 extractive examples are used for training.
  • A shared pre-processing pipeline splits documents into chunks, ranks them by tf-idf similarity to the question, and merges them up to a fixed length: 400 tokens for DocQA and 512 wordpieces for BertQA.
  • Three factors explain much of the generalization gap: differences in context language, differences in question language, and the targeted language phenomenon.
  • Multi-dataset pre-training improves both generalization and transfer. It saves roughly two thirds of target annotation, and a single BERT-large model can perform near or above the state of the art on all datasets without per-dataset fine-tuning.
  • The infrastructure is built on AllenNLP and released openly. Unified-format data is at www.tau-nlp.org/multiqa and code at github.com/alontalmor/multiqa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1485

@inproceedings{talmor2019multiqa,
  title = {{MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension}},
  author = {Talmor, Alon and Berant, Jonathan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1485},
  url = {https://doi.org/10.18653/v1/p19-1485}
}

References

47 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. dataset , even in the presence of BERT representa-text (A DD S ENT variant). M ULTI QA obtained tions. It also leads to substantial reduction in the 66.7 EM and 73.1 F 1 , outperforming BERT-large number of necessary training examples for a fixed (60.4EM, 66.3F1) by a significant margin, and perform… find
  2. Sun et al. (2018) • Performance over an RC dataset can be improved has recently shown this in a larger experiment for by retrieving web snippets for all questions and multi-choice questions, where they first fine-tuned adding them as examples (context augmentation). BERT on RACE (Lai et al., 2017) a… find
  3. J. Berant, V. Srikumar, P. Chen, A. V. Linden, B. Harding, B. Huang, P. Clark, and C. D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  4. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In International Conference on Management of Data (SIGMOD), pages 1247–1250. find
  5. E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. 2018. Quac: Question answering in context. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  6. R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  7. A. Chronopoulou, C. Baziotis, and A. Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. arXiv preprint arXiv:1902.10547. find
  8. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). find our card
  9. Y. Chung, H. Lee, and J. Glass. 2018. Supervised and unsupervised transfer learning for question answering. In North American Association for Computational Linguistics (NAACL). find
  10. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
  11. C. Clark and M. Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Association for Computational Linguistics (ACL). find our card
  12. R. Das, S. Dhuliawala, M. Zaheer, and A. McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations (ICLR). find
  13. Y. Lin, H. Ji, Z. Liu, and M. Sun. 2018. Denoising distantly supervised open-domain question answering. In Association for Computational Linguistics (ACL), volume 1, pages 1736–1745. find
  14. X. Liu, P. He, W. Chen, and J. Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. find our card
  15. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL). find
  16. K. Luo1, F. Lin1, X., L. Kenny, and Q.Zhu1. 2018. Knowledge base question answering via encoding of complex query graphs. In Empirical Methods in Natural Language Processing (EMNLP). find
  17. M. Ding, C. Zhou, Q. Chen, H. Yang, and J. Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In Association for Computational Linguistics (ACL). find
  18. B. McCann, N. S. Keskar, C. Xiong, and R. Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. find our card
  19. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL). find our card
  20. S. Min, M. Seo, and H. Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Association for Computational Linguistics (ACL). find
  21. M. Dunn, , L. Sagun, M. Higgins, U. Guney, V. Cirik, and K. Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv. find our card
  22. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. find our card
  23. T. M. Fruchterman and E. M. Reingold. 1991. Graph drawing by force-directed placement. Software: Practice and experience, 21(11):1129–1164. find
  24. M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, and L. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640. find our card
  25. K. M. Hermann, T. Koisk, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  26. M. Hu, Y. Peng, F. Wei, Z. Huang, D. Li, N. Yang, and M. Zhou. 2018. Attention-guided answer distillation for machine reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). find
  27. J. Pennington, R. Socher, and C. D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  28. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). find
  29. J. Phang, T. Fevry, and S. R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  30. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  31. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  32. Y. Xu, X. Liu, Y. Shen, J. Liu, and J. Gao. 2019. Multi-task learning with sample re-weighting for machine reading comprehension. In North American Association for Computational Linguistics (NAACL). find
  33. S. Reddy, D. Chen, and C. D. Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. find our card
  34. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  35. M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. find our card
  36. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv. find our card
  37. D. Yogatama, C. de M. d’Autume, J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazaridou, W. Ling, L. Yu, C. Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. find
  38. K. Sun, D. Yu, D. Yu, and C. Cardie. 2018. Improving machine reading comprehension with general reading strategies. arXiv preprint arXiv:1810.13441. find
  39. A. Talmor and J. Berant. 2018a. Repartitioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623. — [arXiv:1807.09623](https://arxiv.org/abs/1807.09623) find
  40. A. Talmor and J. Berant. 2018b. Repartitioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623. — [arXiv:1807.09623](https://arxiv.org/abs/1807.09623) find
  41. A. Talmor and J. Berant. 2018c. The web as knowledge-base for answering complex questions. In North American Association for Computational Linguistics (NAACL). find
  42. A. Talmor, M. Geva, and J. Berant. 2017. Evaluating semantic parsing against a simple web-based question answering model. In *SEM. find
  43. Y. Tay, L. Tuan, S. Hui, and J. Su. 2018. Densely connected attention propagation for reading comprehension. In Advances in Neural Information Processing Systems (NeurIPS). find
  44. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP. find our card
  45. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR). find our card
  46. W. Wang, M. Yan, and C. Wu. 2018. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. In Association for Computational Linguistics (ACL). find
  47. J. Welbl, P. Stenetorp, and S. Riedel. 2017. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/