MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-5801.pdf

Summary

Reading comprehension models are usually scored on test data that matches their training distribution, so it is unclear how well they cope with unfamiliar domains. The MRQA 2019 Shared Task measures this out-of-domain generalization for English extractive question answering. The organizers converted 18 existing QA datasets into a single span-extraction format. Contexts are capped at 800 tokens and marked with [DOC], [TLE] and [PAR] tokens. Multiple-choice and answer-string-only sources were adapted as needed. The datasets were split three ways: six for training, six for development and testing, and six fully hidden from participants. Systems were ranked by macro-averaged F1 over the 12 held-out test sets. Ten teams took part, trying approaches such as:

• data sampling
• multi-task learning
• adversarial domain training
• ensembling

The winning system, D-Net, built on newer pre-trained language models and reached 72.5 F1, 10.7 points above the BERT-based baseline. The released benchmark gives the community a common, reusable way to test whether QA systems hold up beyond their training data.

Key points

• Training sets (Split I) are SQuAD, NewsQA, TriviaQA, SearchQA, HotpotQA and Natural Questions. Split II includes BioASQ, DROP, DuoRC, RACE, RelationExtraction and TextbookQA. Split III includes BioProcess, ComplexWebQ, MCTest, QAMR, QAST and TREC.
• Test portions were rebalanced to about 1,500 examples per sub-domain, partitioned by context. Scoring uses EM and F1 based on answer-string match.
• Adversarial training with a domain discriminator added +1.9 F1 over a BERT-Base model but only +0.4 F1 over BERT-Large. Negative sampling gave up to +1.5 F1 on Split II.
• Fine-tuning the BERT-Large baseline on Split II development data raised its score by 4.6 F1. This closed the gap to D-Net only on RelationExtraction.