Elamur Research
☰
home / papers / 2019 / ELI5: Long Form Question Answering

ELI5: Long Form Question Answering

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1346.pdf

Summary

Most question answering benchmarks target short or span-extracted answers, so explanatory "how" and "why" questions remain largely unaddressed. This work defines long-form question answering and builds ELI5, a large corpus from the Reddit forum "Explain Like I'm Five." Each question is paired with the top-voted community answer, filtered by vote scores, and with evidence from Common Crawl. For every question, 100 web pages are retrieved with Apache Lucene and condensed through TFIDF-based passage selection into a single support document. The authors compare extractive approaches (including a BidAF-based model adapted to multi-sentence output), language models, and Seq2Seq models. The strongest system is a Seq2Seq model trained jointly on language modeling, masked word prediction, and answer generation, which outperforms the other baselines in both ROUGE and human judgments. Human-written answers still clearly win, and analysis shows that the compressed evidence omits much of the useful information. ELI5 therefore provides a testbed for retrieval over long inputs and for generating coherent paragraph-length answers.

Key points

  • ELI5 contains about 272K question–answer pairs, split into 237K train, 10K validation, and 25K test. The validation and test questions were chosen so that none has a close TFIDF neighbor in training.
  • Average lengths are roughly 42 words per question, 858 words per support document, and 131 words per answer, about an order of magnitude longer than answers in SQuAD, TriviaQA, or MS MARCO.
  • On a manually annotated subset, 65% of support documents contained the answer and 92% held relevant information. In 63% of threads, at least one additional answer meets the vote threshold.
  • In pairwise human evaluation, raters chose the gold answer over every model in at least 85.5% of cases. The multi-task Seq2Seq model was preferred to the extractive baseline in only 57% of comparisons.
  • Oracle experiments show that fewer than 9% of the best-matching sentences come from the top 20 retrieved passages. Training on more data steadily raises ROUGE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

365 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 330 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1346 · arXiv 1907.09190

@inproceedings{fan2019eli5,
  title = {{ELI5: Long Form Question Answering}},
  author = {Fan, Angela and Jernite, Yacine and Perez, Ethan and Grangier, David and Weston, Jason and Auli, Michael},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1346},
  eprint = {1907.09190},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1346}
}

References

27 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146. find
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. find our card
  3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In EMNLP. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  5. Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/1704.05179. find our card
  6. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In ACL Workshop on Neural Machine Translation and Generation. find our card
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML. find our card
  8. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. CoRR, abs/1702.08734. find our card
  9. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. find our card
  10. Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. TACL. find our card
  11. Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In ACL Workshop on Text Summarization Branches Out. find
  12. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In ICLR. find our card
  13. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. CoRR. find our card
  14. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT, pages 1–9. Association for Computational Linguistics. find our card
  15. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  16. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  17. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In ACL. find our card
  18. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  19. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. find our card
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. find
  21. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. find our card
  22. Anastasios Tombros and Mark Sanderson. 1998. Advantages of query biased summaries in information retrieval. In SIGIR. find
  23. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In ACL Workshop on Representation Learning for NLP. find our card
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS. find
  25. Ellen M. Voorhees. 2003. Overview of the TREC 2003 question answering track. In TREC. find
  26. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural qa as simple as possible but not simpler. In CoNLL. find
  27. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. find our card