📄 Who did What: A Large-Scale Person-Centered Cloze Dataset

Summary

Large cloze-style reading comprehension benchmarks such as CNN/Daily Mail build questions from article summaries and anonymize entities. This often lets models succeed through surface matching or frequency heuristics. Onishi et al. address this with Who-did-What (WDW), a set of over 200,000 multiple-choice problems drawn from the LDC English Gigaword newswire corpus.

Each problem pairs two separate articles. A person mention is blanked out of the opening sentence of one article, and Apache Lucene retrieves a different article covering the same events, published within two weeks, to serve as the passage. Answer candidates are the real, non-anonymized person names found in that passage. The authors then solve an optimization that discards items easily answered by four simple heuristics, pushing those heuristics down to chance level.

Humans still answer 84% correctly, yet established neural readers score notably lower than on CNN. The resulting human–machine gap makes WDW a harder test of semantic understanding. Because it needs no human-written summaries, the construction method can also be applied to other news collections.

Key points

• Construction pipeline: Stanford NER and the Stanford PCFG parser find person mentions, and each blank removes the entire noun phrase headed by the person, so descriptors cannot give the answer away. Passages that overlap the question too closely are filtered out, and only problems with 2–5 choices are kept.

• Baseline suppression: The four suppressed baselines are first person in the passage, most frequent person, a Gigaword 5-gram language model, and unigram last-name frequency. Suppression removed 49.9% of the problems, and every baseline fell from 0.43–0.61 accuracy to about 0.32, which is random level.

• Splits:

  • train: 127,786 problems
  • "relaxed" train (less suppression): 185,978 problems
  • validation: 10,000 problems
  • test: 10,000 problems

  Validation and test come from the most recent articles, which limits topical overlap with training. Problems average about 3.4–3.5 choices.

• Human vs. model performance: Human accuracy is 84%, compared with roughly 75% on CNN and 82% on CBT named entities.

  • Attentive Reader drops from 63% on CNN to 55% on WDW.
  • Attention Sum Reader drops from 70% to 59%.
  • Readers that exploit answer frequency (AS, GA) lose up to 17 points.

• Availability: The dataset is distributed via tticnlp.github.io/who_did_what.