Elamur Research⌕
☰
home / papers / 2021 / End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering

End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering

Paper · 2021 · Devendra Singh Sachan, Siva Reddy, William Hamilton et al.

Summary

Open-domain question answering systems usually pair a retriever, which finds evidence passages in a large corpus such as Wikipedia, with a reader that produces the answer. Training both parts jointly is hard when the reader fuses many passages at once, because the set of retrieved documents acts as a latent variable that cannot be marginalized exactly. Earlier systems either train the components in alternating stages or restrict the reader to one document at a time. The authors propose EMDR², which treats retrieval as a latent set and applies an expectation-maximization-style approximation. A BERT-sized dual-encoder retriever selects the top-K passages, and a Fusion-in-Decoder reader built on T5-base generates the answer. The reader learns from the retriever's prior top-K choices. The retriever learns from an approximate posterior that also uses the gold answer. On Natural Questions, TriviaQA and WebQuestions, EMDR² beats comparably sized models by about 2–3 exact-match points. It works well even with an unsupervised retriever initialization, which suggests that labelled retrieval supervision may be unnecessary.

Key points

  • Architecture: a dual-encoder retriever (12-layer, 768-hidden, BERT-like transformers using the [CLS] embedding and dot-product scoring) plus a Fusion-in-Decoder reader on T5-base. Each passage is encoded with the question and its title, at N = 512 tokens per passage.
  • Training signal: the reader is trained on top-K documents chosen by prior retrieval scores, so training matches inference. The retriever is updated toward an approximate posterior that takes the answer into account.
  • Development-set exact match with top-50 documents (FiD baseline vs. EMDR²):
    • NQ: 47.3 vs. 50.4
    • TriviaQA: 65.5 vs. 71.1
    • WebQ: 46.0 vs. 49.9
  • Ablation of alternative objectives: an adversarial variant (L_alt-1) collapses, reaching only 14.1 EM on NQ. A uniform-prior variant (L_alt-2) beats FiD on NQ and TriviaQA but diverges on WebQ.
  • Limitations and release: training needs 16 GPUs, and the uncompressed evidence index is costly to store and search. Code and checkpoints are released at github.com/DevSinghSachan/emdr2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🎓McGill UniversityMontreal · Canada91 works · 11,178 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada65 works · 14,425 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 487 worksInformation retrieval & search · 552 works

Cite

Cite as Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer and Dani Yogatama. 2021. End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering. arXiv:2106.05346.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.05346 · arXiv 2106.05346

@misc{sachan2021end,
  title = {{End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering}},
  author = {Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer and Dani Yogatama},
  year = {2021},
  doi = {10.48550/arxiv.2106.05346},
  eprint = {2106.05346},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.05346}
}

References

37 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berant, J., Chou, A., Frostig, R., and Liang, P. (2013). Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.
  2. Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., and Shah, R. (1994). Signature verification using a "siamese" time delay neural network. In Advances in Neural Information Processing Systems. find
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… — Language Models are Few-Shot Learners
  4. Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017). Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  5. Clark, C. and Gardner, M. (2018). Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). de Masson d’Autume, C., Ruder, S., Kong, L., and Yogatama, D. (2019). Episodic memory in l…
  6. Dempster, A., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, Series B, 1(39), 1–38. find
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (…
  8. Fan, A., Gardent, C., Braud, C., and Bordes, A. (2021). Augmenting Transformers with KNN-Based Composite Memory for Dialog. Transactions of the Association for Computational Linguistics, 9.
  9. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning.
  10. Iyer, S., Min, S., Mehdad, Y., and Yih, W. (2021). Reconsider: Re-ranking using span-focused cross-attention for open domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1…
  11. Izacard, G. and Grave, E. (2021a). Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations. find
  12. Izacard, G. and Grave, E. (2021b). Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.
  13. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  14. Karpukhin, V., Oğuz, B., Min, S., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  15. Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. (2020). Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  16. Khandelwal, U., Fan, A., Jurafsky, D., Zettlemoyer, L., and Lewis, M. (2021). Nearest neighbor machine translation. In International Conference on Learning Representations. find
  17. Khattab, O. and Zaharia, M. (2020). Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
  18. Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations.
  19. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: a benchmark for question answering res…
  20. Lee, K., Chang, M.-W., and Toutanova, K. (2019). Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
  21. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020a). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Comp…
  22. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020b). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems.
  23. Luan, Y., Eisenstein, J., Toutanova, K., and Collins, M. (2021). Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9.
  24. Min, S., Chen, D., Hajishirzi, H., and Zettlemoyer, L. (2019). A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM…
  25. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, hig…
  26. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. (2019). Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM…
  27. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. (2020). Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
  28. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67.
  29. Roberts, A., Raffel, C., and Shazeer, N. (2020). How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  30. Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval. find
  31. Sachan, D. S., Patwary, M., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., and Catanzaro, B. (2021). End-to-end training of neural retrievers for open-domain question answering. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio…
  32. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. find
  33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems.
  34. Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., and Jiang, J. (2018). R3: Reinforced ranker-reader for open-domain question answering. In AAAI. find
  35. Wang, Z., Ng, P., Ma, X., Nallapati, R., and Xiang, B. (2019). Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang…
  36. Yih, W.-t., Toutanova, K., Platt, J. C., and Meek, C. (2011). Learning discriminative projections for text similarity measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning.
  37. Yogatama, D., de Masson d’Autume, C., and Kong, L. (2021). Adaptive Semiparametric Language Models. Transactions of the Association for Computational Linguistics, 9, 362–373. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sachan-2021-end-end-training-multi-document/