RACE: Large-scale ReAding Comprehension Dataset From Examinations
What it is about
RACE collects about 28,000 passages and 100,000 four-option questions from English exams for Chinese middle-school (RACE-M, ages 12–15) and high-school (RACE-H) students, written by teachers; options are human-written sentences, often not in the passage. On 500 annotated questions, only 15.8% are word matching, 19.2% paraphrase, 33.4% single-sentence reasoning, 25.8% multi-sentence reasoning and 5.8% insufficient or ambiguous — far more reasoning than SQuAD or CNN. Reasoning questions are further split into detail reasoning, whole-picture understanding, passage summarisation («The main idea is…»), attitude analysis («The author's attitude toward X is…») and world knowledge (arithmetic, dates). A sliding-window word-matching baseline gets 32%, neural readers about 44%, crowd workers 73%, and the ceiling (questions with an unambiguous answer) 94.5%.
Ideas
- I1. Add the sliding-window baseline of Richardson et al. (the one used here: TF-IDF overlap of question + option with every window of the lesson) to Mova's course reports as the floor every course must beat, per course and per question type. It costs nothing, is deterministic, and shows where the student is no better than word overlap.
- I2. Label each course question with the paper's difficulty classes by rule: word match (all content lemmas of question+right option in one sentence), paraphrase (in one sentence after synonym/lemma mapping), single-sentence, multi-sentence (frames from two or more sentences needed — the decision trace shows how many frames supported the answer), insufficient (no supporting frame at all). Report the student's accuracy per class next to
world knowledge diagnose; the «insufficient» share is the ceiling the Torah diagnosis already hit (quotes not in the lesson). - I3. Route the two whole-text question types instead of matching them to one frame: «main idea» options compared against the most salient agent and the frames with the most links, «attitude» options against the polarity of the evaluation and express frames of that character (
pragpolarity/act=evaluate). Count first how many course questions are of these shapes.
For Mova
languages/en/30-pragmatics-overview.md §9 (measurements), 32-politeness-and-hedging.md (attitude), 36-discourse-relations.md. Code: world/src/knowledge.rs (diagnosis, question classes), world/src/report.rs (baseline column), world/src/school.rs. Measures: student minus window-baseline per course, accuracy per difficulty class, share of insufficient questions.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-1082.pdf
Summary
Most earlier machine reading comprehension benchmarks take their answers directly from the passage, and their questions are crowdsourced or generated automatically. They also tend to cover only one genre. As a result, models can score well through surface matching rather than understanding. The authors built RACE from English exams that teachers wrote for Chinese middle- and high-school students aged 12–18. Each passage has several multiple-choice questions with four options, and neither the questions nor the options have to be spans copied from the text. The passages come from many genres, including news, stories, ads and biographies. The dataset is split into RACE-M (middle school) and RACE-H (high school). The authors tested a sliding-window matcher and two neural readers, Stanford AR and Gated AR. They estimated the human ceiling by checking how many questions are valid. The neural models reached only about 43–44% accuracy, while the ceiling is close to 95%. RACE also has a larger share of questions that need reasoning than earlier corpora. This makes it a hard, clean benchmark that is still large enough to train deep models.
Key points
- RACE contains 27,933 passages and 97,687 questions written by English teachers. Every question has four options and exactly one correct answer. The data and baseline code are freely available.
- It adds question types that no earlier large-scale dataset had, such as summarizing a passage and identifying an author's attitude. Answer options are free-form sentences, not text spans.
- Stanford AR scored 43.3% and Gated AR scored 44.1%. These same model types perform much better on CNN/Daily Mail, CBT and Who-did-What.
- The sliding-window baseline reached 51.5% on MCTest but only 37.3% on RACE. Its gain over random guessing was 28.6 points on RACE, compared with 50–92 points on CBT and WDW, so far fewer RACE questions can be solved by simple matching.
- About 94.5% of questions are valid, which sets the human ceiling (95.4% for RACE-M, 94.2% for RACE-H). Amazon Mechanical Turk workers scored 85% on RACE-M and 70% on RACE-H.
Cite
@inproceedings{lai2017race,
title = {{RACE: Large-scale ReAding Comprehension Dataset From Examinations}},
author = {Lai, Guokun and Xie, Qizhe and Liu, Hanxiao and Yang, Yiming and Hovy, Eduard},
booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
year = {2017},
doi = {10.18653/v1/d17-1082},
eprint = {1704.04683},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d17-1082}
}References
- Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956 . — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
- Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858 . find our card
- Bhuwan Dhingra, Hanxiao Liu, William W Cohen, and Ruslan Salakhutdinov. 2016. Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549 . find
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701. find our card
- Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301 . find our card
- Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547 . find
- Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. arXiv preprint arXiv:1604.06076 . — [arXiv:1604.06076](https://arxiv.org/abs/1604.06076) find
- Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Lingu… find
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 . find our card
- Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457 . find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
- Anselmo Peñas, Yusuke Miyao, Álvaro Rodrigo, Eduard H Hovy, and Noriko Kando. 2014. Overview of clef qa entrance exams task 2014. In CLEF (Working Notes). pages 1194–1200. find
- Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 . find our card
- Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP. volume 3, page 4. find our card
- Álvaro Rodrigo, Anselmo Peñas, Yusuke Miyao, Eduard H Hovy, and Noriko Kando. 2015. Overview of clef qa entrance exams task 2015. In CLEF (Working Notes). find
- Hideyuki Shibuki, Kotaro Sakamoto, Yoshinobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y Itakura, Di Wang, Tatsunori Mori, and Noriko Kando. 2014. Overview of the ntcir-11 qa-lab task. In NTCIR. find
- Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 . find our card
- Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. arXiv preprint arXiv:1702.02206 . find