NewsQA: A Machine Comprehension Dataset

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-2623.pdf

Summary

Before NewsQA, large machine comprehension (MC) datasets were either too small for deep learning or generated synthetically. Even SQuAD, which uses crowdsourced questions, was already close to human-level scores. The Microsoft Maluuba team built NewsQA, a span-extraction corpus of 119,633 crowdsourced questions about 12,744 CNN articles. The people who wrote the questions saw only each article's headline and highlights. This led to curiosity-driven questions that use different wording from the article, and some of them cannot be answered from it. A separate group of workers then marked answer spans in the full article, and a validation round settled disagreements. Compared with SQuAD, the documents are much longer, and more questions need inference or need facts combined from several sentences. Humans scored 69.4% F1. The best baseline, FastQA, was 13.3 points lower, while the same human–model gap on SQuAD was under 2 points. NewsQA is freely available and is a harder benchmark that complements SQuAD. It can be used to test cross-domain transfer and to train models to recognise questions without an answer.

Key points

• Collection pipeline: articles were sampled at random from 90,266 CNN articles and split 90/5/5 into train/dev/test. Each question was answered by 2.73 workers on average. About 43.2% of questions went to validation, and 86.0% end up with an answer that at least two workers agreed on.
• Unanswerable questions: questions without an agreed answer are kept in the dataset and marked. Answerers could also choose a null answer.
• Document length: NewsQA articles average 30.7 sentences, compared with 4.9 for SQuAD paragraphs. A simple isf sentence-selection heuristic finds the answer sentence 79.6% of the time on SQuAD but only 35.4% on NewsQA. The gap stays large even when SQuAD paragraphs are joined into longer documents.
• Human scores: on 1,000 dev questions, humans scored 69.4 F1, 46.5 EM, 56.0 BLEU and 3.596 CIDEr. The authors say the low EM partly reflects equally correct answers with different span boundaries.
• Neural baselines: FastQA beat the authors' mLSTM on both datasets. Model errors were highest on synthesis and inference questions, and the model handled named-entity answers best.