📄 XOR QA: Cross-lingual Open-Retrieval Question Answering

Summary

Most multilingual QA benchmarks assume the answer is written in the same language as the question. In practice, many languages have few relevant articles (information scarcity), and native speakers often ask about foreign entities that local Wikipedias barely cover (information asymmetry). The authors define XOR QA, an open-retrieval setting where a question in one language may be answered using documents in another language. They also build XOR-TyDi QA, a dataset of about 40K questions in 7 typologically diverse languages. The questions come from TyDi QA questions that had no same-language answer. Professional translators rendered them into English. Crowdworkers then located answers in English Wikipedia with help from a model-assisted ranking setup, and the answers were verified and translated back. The authors propose three tasks of increasing difficulty and test pipelines that combine machine translation, retrieval and DPR-based reading. The strongest end-to-end system reaches only 18.7 F1 on average, which shows the task is far from solved. Searching both English and native-language collections clearly outperforms using either collection alone.

Key points

• Dataset and languages: XOR-TyDi QA spans seven languages: Telugu, Bengali, Arabic, Russian, Finnish, Korean, and Japanese. Questions were written by native speakers and were originally unanswerable in TyDi QA.
• Annotation pipeline:
  • Questions were translated via Gengo; native-speaker checks rated over 95% of sampled translations as correct.
  • Annotators on Amazon Mechanical Turk worked with Google Search results (up to 10 articles) re-ranked by Path Retriever.
  • Roughly 70% of answers came from the five top-ranked paragraphs, and the rest from deeper reading.
• Three tasks:
  • XOR-Retrieve: find English passages that answer the question.
  • XOR-English Span: extract the answer span from English text.
  • XOR-Full: produce the answer in the question's language using both Wikipedias.
• Best XOR-Full result: the pipeline of Google Translate, Google Search and DPR scores 18.7 F1, 12.1 EM and 16.8 BLEU (macro average). Replacing it with open MT and BM25 causes large drops, for example Telugu falls to 1.7 F1.
• Single-collection ablation: restricting retrieval to only English or only native-language Wikipedia generally hurts performance, which supports multilingual retrieval.