📄 NewsQA: A Machine Comprehension Dataset

Summary

Before NewsQA, large machine comprehension (MC) datasets were either too small for deep learning or generated synthetically. Even SQuAD, which uses crowdsourced questions, was already close to human-level scores. The Microsoft Maluuba team built NewsQA, a span-extraction corpus of 119,633 crowdsourced questions about 12,744 CNN articles. The people who wrote the questions saw only each article's headline and highlights. This led to curiosity-driven questions that use different wording from the article, and some of them cannot be answered from it. A separate group of workers then marked answer spans in the full article, and a validation round settled disagreements. Compared with SQuAD, the documents are much longer, and more questions need inference or need facts combined from several sentences. Humans scored 69.4% F1. The best baseline, FastQA, was 13.3 points lower, while the same human–model gap on SQuAD was under 2 points. NewsQA is freely available and is a harder benchmark that complements SQuAD. It can be used to test cross-domain transfer and to train models to recognise questions without an answer.

Key points

• Collection pipeline: articles were sampled at random from 90,266 CNN articles and split 90/5/5 into train/dev/test. Each question was answered by 2.73 workers on average. About 43.2% of questions went to validation, and 86.0% end up with an answer that at least two workers agreed on.
• Unanswerable questions: questions without an agreed answer are kept in the dataset and marked. Answerers could also choose a null answer.
• Document length: NewsQA articles average 30.7 sentences, compared with 4.9 for SQuAD paragraphs. A simple isf sentence-selection heuristic finds the answer sentence 79.6% of the time on SQuAD but only 35.4% on NewsQA. The gap stays large even when SQuAD paragraphs are joined into longer documents.
• Human scores: on 1,000 dev questions, humans scored 69.4 F1, 46.5 EM, 56.0 BLEU and 3.596 CIDEr. The authors say the low EM partly reflects equally correct answers with different span boundaries.
• Neural baselines: FastQA beat the authors' mLSTM on both datasets. Model errors were highest on synthesis and inference questions, and the model handled named-entity answers best.