Elamur Research⌕
☰
home / papers / 2019 / Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering

Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering

Summary

Open-domain question answering requires finding answers in large collections such as Wikipedia, not in a single given paragraph. Earlier BERT-based systems scored each retrieved passage separately, so answer scores from different passages could not be reliably compared. The authors (AWS AI Labs) propose multi-passage BERT. It encodes each question–passage pair as BERT-RC does, but applies one softmax over the start and end positions of all passages retrieved for a question. They also study several design choices on OpenSQuAD:

  • passage granularity,
  • overlapping sliding-window splitting,
  • a BERT-based passage ranker whose scores are combined with span probabilities,
  • whether explicit question–passage matching (BERT+QANet) still helps.

Global normalization lets accuracy keep rising as more passages are added, whereas per-passage normalization degrades. With all components, the model beats prior systems on Quasar-T, SearchQA, TriviaQA and OpenSQuAD. The work shows that cross-passage score calibration, rather than additional matching layers, is key when adapting reading-comprehension models to retrieval-based QA.

Key points

  • Passage length: 100-word passages work better than single sentences or 50- and 200-word chunks. A sliding window (size 100, stride 50) adds 4.7 EM on the OpenSQuAD validation set (35.7 → 40.4).
  • Passage ranker: The ranker reranks the top-100 BM25/ElasticSearch passages and keeps the top 30. With its scores, validation results rise to 42.8 EM / 53.4 F1 (from 40.4 / 49.8).
  • Explicit matching: Adding QANet-style inter-sentence matching on top of BERT hurts performance. The best variant reaches 36.2 EM. The authors suggest BERT self-attention already performs this matching.
  • OpenSQuAD test results: Multi-passage BERT (Base) reaches 51.2 EM / 59.0 F1, and the Large version reaches 53.0 / 60.9. This is +21.4 EM over the best non-BERT system and +5.8 EM over the authors' BERT-RC baseline.
  • Other benchmarks (Large): TriviaQA 63.7 / 69.2, SearchQA 65.1 / 70.7, Quasar-T 51.1 / 59.1 (EM / F1).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1599

@inproceedings{wang2019multi,
  title = {{Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering}},
  author = {Wang, Zhiguo and Ng, Patrick and Ma, Xiaofei and Nallapati, Ramesh and Xiang, Bing},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1599},
  url = {https://doi.org/10.18653/v1/d19-1599}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. 2018. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks. arXiv preprint arXiv:1809.02040. — [arXiv:1809.02040](https://arxiv.org/abs/1809.02040) find
  2. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) find
  3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL 2017. find our card
  4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL 2018. find our card
  5. Mostafa Dehghani, Hosein Azarbonyad, Jaap Kamps, and Maarten de Rijke. 2019. Learning to transform, combine, and reason in open-domain question answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 681–689. ACM. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  7. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
  8. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find our card
  9. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL 2017. find our card
  10. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In ACL 2018. find
  11. Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In ICLR 2017. find our card
  12. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R3: Reinforced ranker-reader for open-domain question answering. In AAAI 2018. find
  13. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR 2018. find
  14. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In ACL 2017. find our card
  15. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211. — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) find
  16. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718. find our card
  17. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR 2018. find our card
  18. Rodrigo Nogueira, Jannis Bulian, and Massimiliano Ciaramita. 2018. Learning to coordinate multiple reinforcement learning agents for diverse query reformulation. arXiv preprint arXiv:1809.10658. — [arXiv:1809.10658](https://arxiv.org/abs/1809.10658) find
  19. Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Lixin Su, and Xueqi Cheng. 2019. Has-qa: Hierarchical answer spans model for open-domain question answering. In AAAI 2019. find
  20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP 2016. find our card
  21. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR 2017. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2019-multi-passage-bert-globally-normalized/