Elamur Research⌕
☰
home / papers / 2018 / Denoising Distantly Supervised Open-Domain Question Answering

Denoising Distantly Supervised Open-Domain Question Answering

Paper · 2018 · Yankai Lin, Haozhe Ji, Zhiyuan Liu et al.

Summary

Open-domain QA systems trained with distant supervision (DS-QA) find candidate paragraphs through retrieval and then run a reading-comprehension model over them. This setup has two weaknesses. Many retrieved paragraphs, and many answer-string matches inside them, are wrongly labeled. Earlier systems also usually read only the single top paragraph and discard useful evidence elsewhere. The authors address both problems with a coarse-to-fine model. First, a lightweight paragraph selector (MLP or single-layer BiLSTM encoders, a self-attentive question vector, max-pooling and softmax) assigns a relevance probability to every retrieved paragraph. Second, a multi-layer BiLSTM paragraph reader predicts answer spans within each paragraph. The answer probability is obtained by marginalizing over paragraphs, weighted by the selector's scores, and both parts are trained jointly. Experiments on Quasar-T, SearchQA and TriviaQA show consistent gains over baselines, including R^3. The selector is much better than plain IR at ranking useful paragraphs, and it lets the system read only about half of the paragraphs without losing accuracy. This makes DS-QA both more robust to noise and faster.

Key points

  • The model combines a fast paragraph selector that filters noise with a precise paragraph reader, then aggregates answers across all informative paragraphs instead of using only the best one. Code and data are released at github.com/thunlp/OpenQA.
  • On paragraph selection, Hits@1 rises from 6.3 with IR to 27.7 with Our+FULL on Quasar-T, and from 13.7 to 58.9 on SearchQA.
  • In the same comparison, Our+FULL reaches EM/F1 of 41.1/48.0 on Quasar-T and 58.8/64.5 on SearchQA. Joint training (Our+FULL) beats independent training (Our+INDEP).
  • On both datasets, keeping the top half of paragraphs chosen by the selector causes no performance drop, whereas IR-based truncation degrades clearly.
  • In a re-ranking upper-bound analysis on the dev sets, Our+FULL top-1 EM exceeds R^3 (42.2 vs 35.3 on Quasar-T; 58.8 vs 51.2 on SearchQA). Top-3/5 scores sit 10–20% above top-1, which indicates room for answer re-ranking.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🔬State Key Laboratory of Intelligent Technology and SystemsBeijing · China21 works · 9,641 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 242 works

Cite

Cite as Yankai Lin, Haozhe Ji, Zhiyuan Liu and Maosong Sun. 2018. Denoising Distantly Supervised Open-Domain Question Answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1161

@inproceedings{lin2018denoising,
  title = {{Denoising Distantly Supervised Open-Domain Question Answering}},
  author = {Yankai Lin, Haozhe Ji, Zhiyuan Liu and Maosong Sun},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1161},
  url = {https://doi.org/10.18653/v1/p18-1161}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Table 6 shows two examples of our models, which illustrates that our model can make full use of informative paragraphs. From the table we find that: (1) For the question “Who directed the 1946 ‘It’s A Wonderful Life’?”, our model extracts the answer “Frank Capra” from both top-2 paragraphs ranked by… find
  2. Conclusion and Future Work Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013a. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP. pages 1533–1544.
  3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013b. Semantic parsing on Freebase from question-answer pairs. In Proceedings of EMNLP. pages 1533–1544.
  4. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075 . find
  5. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830 . find
  6. In this paper, we propose a denoising distantly supervised open-domain question answering system which contains a paragraph selector to skim over paragraphs and a paragraph reader to perform an intensive reading on the selected paragraphs. Our model can make full use of all informative paragraphs an… find
  7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of ACL. pages 2358–2367. \#\# Acknowledgments Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017b. Quasar: Datasets for question answering by sea…
  8. This work is supported by the National Natural Science Foundation of China (NSFC No. 61572273, 61661146007 and 61572273). This paper is also partially funded by Microsoft Research Asia FY17-RES-THEME-017. find
  9. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the ACL. pages 1870–1879.
  10. Tongfei Chen and Benjamin Van Durme. 2017. Discriminative information retrieval for question answering sentence selection. In Proceedings of EACL. pages 719–725. find
  11. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of ACL. pages 484–494.
  12. Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of ACL. pages 209– 220. find
  13. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In Proceedings of ACL. pages 593–602. find
  14. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017a. Gated-attention readers for text comprehension. In Proceedings of ACL. pages 1832–1846. find
  15. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179 .
  16. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of SIGKDD. pages 1156–1165. find
  17. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL. pages 1480–1489.
  18. Bert F Green Jr, Alice K Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: an automatic question-answerer. In Proceedings of IRE-AIEE-ACM. pages 219–224. find
  19. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL. pages 1601–1611.
  20. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  21. Cody Kwok, Oren Etzioni, and Daniel S Weld. 2001. Scaling question answering to the web. TOIS pages 242–262. find
  22. Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of ACL. pages 2124–2133. find
  23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP. pages 1532–1543.
  24. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of ICLR.
  25. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of SIGKDD. ACM, pages 1047–1055. find
  26. Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Proceedings of TREC. pages 77–82.
  27. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of AAAI. find
  28. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In Proceedings of ICLR. find
  29. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of ACL. pages 189–198.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2018-denoising-distantly-supervised-open-domain/