TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.30.pdf

Summary

TyDi QA is a multilingual question answering benchmark built from real information-seeking questions. It addresses a gap in multilingual evaluation: most QA datasets are English-only, and multilingual ones like MLQA and XQuAD are mostly translated from English. Translation leaves artifacts ("Translationese") and inherits English topics. Lexical overlap between question and answer can also make these datasets artificially easy. For TyDi QA, annotators in each language read a short Wikipedia opening snippet and wrote questions they actually wanted answered but couldn't answer from the snippet. A Google search then paired each question with a Wikipedia article, and annotators marked a passage answer and a minimal answer span, YES/NO, or no answer. No translation or model was used at any stage. The resulting 204K question-answer pairs cover 11 typologically diverse languages and have much less lexical overlap than MLQA or XQuAD. Even a simplified gold-passage version leaves mBERT well below human performance. Models trained on SQuAD transfer poorly to it. The benchmark gives a more realistic way to test whether models generalize across the world's languages.

Key points

• Size and languages: 204K question-answer pairs in 11 languages, including Arabic, Bengali, Finnish, Indonesian, Kiswahili, Korean, Russian, Telugu, Thai, Japanese and English. Data, an open-source baseline and a leaderboard with a hidden test set are public (github.com/google-research-datasets/tydiqa).
• Two primary tasks: Passage Selection (pick the answering paragraph or NULL) and Minimal Answer Span (byte span, YES/NO, or NULL). Many questions have no answer, which happens naturally rather than by design.
• Simplified Gold Passage task (GoldP): only the gold passage is given, unanswerable questions are dropped, scoring uses SQuAD 1.1 metrics, and Thai and Japanese are excluded. It is compatible with code written for SQuAD, XQuAD and MLQA.
• GoldP F1 results: mBERT fine-tuned on TyDi QA scores 79.0 overall, SQuAD-only zero-shot transfer scores 56.4, and estimated human performance is 90.9. Even in English, the SQuAD-trained model scores lower (73.4 vs 76.8 for the TyDi QA-trained model).
• No cross-language comparison: each language has its own question set, so the authors warn against comparing scores between languages. They also release no machine translations, to leave room for MT-based research.