Elamur Research⌕
☰
home / papers / 2017 / Making Neural QA as Simple as Possible but not Simpler

Making Neural QA as Simple as Possible but not Simpler

Paper · 2017 · Dirk Weissenborn, Georg Wiese and Laura Seiffe

Summary

Large extractive QA datasets such as SQuAD and NewsQA prompted many neural reading models. These models grew steadily more elaborate, especially in their layers for word-by-word interaction between question and context, yet they were seldom compared against a strong simple baseline. Weissenborn, Wiese and Seiffe build such baselines from an intuitive context/type matching heuristic: a correct span should have the answer type the question implies and should appear near many question words.

They first build a bag-of-words model that follows this rule almost literally. They then introduce FastQA, which passes context embeddings and two word-in-question features (one binary, one embedding-weighted) through a single BiRNN and predicts answer start and end separately. FastQA has no dedicated interaction layer, yet it performs competitively against much more complex systems. Adding an interaction layer (FastQAExt) yields only small, unsystematic gains at a clear cost in speed and memory.

The study suggests that two ingredients are enough for strong performance: awareness of question words and a composition function richer than bag-of-words. This reframes both earlier architectural complexity and how hard these benchmarks really are.

Key points

  • BoW baseline design: The bag-of-words model scores candidate spans of up to 10 words. Its type score compares a span representation with the lexical answer type (LAT). Its context score combines 12 weighted word-in-question averages, taken over 5-, 10- and 20-token windows on each side of the span.
  • FastQA architecture: Embeddings combine word vectors with a character CNN (filter width 5, max-pooling). A projection and highway layer follow, then a BiRNN that receives the wiq-b and wiq-w features as extra inputs.
  • Efficiency: FastQA runs about twice as fast as FastQAExt and DCN. It needs roughly 2–4× less memory than those two models.
  • FastQA vs. FastQAExt on SQuAD dev: FastQAExt alone answered 589 questions correctly, against 415 for FastQA alone. The authors found no systematic difference between these two sets in question type, length or required reasoning.
  • Error analysis: About 64% of the 55 inspected errors (35) stem directly from applying the heuristic. About 88% of 50 correct answers (44) are also explained by it. Remaining weaknesses include fine-grained answer types, coreference, abbreviations and nested syntax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany79 works · 29,970 citations🔬Hasso Plattner InstitutePotsdam · Germany14 works · 5,514 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Dirk Weissenborn, Georg Wiese and Laura Seiffe. 2017. Making Neural QA as Simple as Possible but not Simpler. arXiv:1703.04816.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.04816 · arXiv 1703.04816

@misc{weissenborn2017making,
  title = {{Making Neural QA as Simple as Possible but not Simpler}},
  author = {Dirk Weissenborn, Georg Wiese and Laura Seiffe},
  year = {2017},
  doi = {10.48550/arxiv.1703.04816},
  eprint = {1703.04816},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.04816}
}

References

26 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man, Rajat Mon… — TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
  2. Andrea Andrenucci and Eriks Sneiders. 2005. Automated question answering: Review of the main approaches. In ICITA. find
  3. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A Thorough Examination of the CNN / Daily Mail Reading Comprehension Task. ACL .
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. ACL .
  5. Hoa Trang Dang, Diane Kelly, and Jimmy J Lin. 2007. Overview of the TREC 2007 Question Answering Track. TREC . find
  6. D. A. Ferrucci. 2012. Introduction to ”This is Watson”. IBM Journal of Research and Development . find
  7. Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a Bayesian Approximation : Representing Model Uncertainty in Deep Learning. ICML . find
  8. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. NIPS .
  9. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. ICLR .
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. LONG SHORT-TERM MEMORY. Neural Computation . find
  11. Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. ICLR .
  12. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. NAACL .
  13. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering. arXiv:1607.06275v1 [cs.CL] . — [arXiv:1607.06275](https://arxiv.org/abs/1607.06275) find
  14. Dan I. Moldovan, Sanda M. Harabagiu, Marius Pasca, Rada Mihalcea, Richard Goodrum, Roxana Girju, and Vasile Rus. 1999. Lasso: A tool for surfing the answer net. In TREC. find
  15. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms Marco: a Human Generated Machine Reading Comprehension Dataset. NIPS .
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation. EMNLP .
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP .
  18. Matthew Richardson, Christopher J C Burges, and Erin Renshaw. 2013. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. EMNLP .
  19. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2017. Bi-Directional Attention Flow for Machine Comprehension. In ICLR. find
  20. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2016. ReasoNet: Learning to Stop Reading in Machine Comprehension. arXiv:1609.05284 . — [arXiv:1609.05284](https://arxiv.org/abs/1609.05284) find
  21. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A Machine Comprehension Dataset. arXiv:1611.09830 .
  22. Shuohang Wang and Jing Jiang. 2017. Machine Comprehension Using Match-LSTM and Answer Pointer. In ICLR.
  23. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2017. Multi-Perspective Context Matching for Machine Comprehension. arXiv:1612.04211 . — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) find
  24. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic Coattention Networks for Question Answering. ICLR .
  25. Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W Cohen, and Ruslan Salakhutdinov. 2017. Words or Characters? Fine-grained Gating for Reading Comprehension. ICLR . find
  26. Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. 2017. End-to-End Reading Comprehension with Dynamic Answer Chunk Ranking. In ArXiv. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weissenborn-2017-making-neural-qa-simple-possible/