📄 The TREC-8 Question Answering Track

Summary

This paper reports on the TREC-8 Question Answering track, the first large-scale shared evaluation of open-domain question answering. NIST gave participants about 1.5 GB of mostly newspaper text and 200 short factual questions, each answerable from at least one document. Systems returned five ranked answer strings per question, limited to 50 or 250 bytes. Human assessors judged whether each string answered the question, and runs were scored by mean reciprocal rank (MRR). The best systems answered more than two thirds of the questions. Most participants predicted the expected answer type, retrieved candidate passages, and looked for nearby entities of that type. Plain bag-of-words passage retrieval was competitive with 250-byte answers but fell behind at 50 bytes, where deeper linguistic processing paid off. The authors also tested the evaluation method itself. Assessors genuinely disagreed about correctness, yet system rankings stayed stable across different judges. This supports single-assessor judging for comparing QA systems, as long as the test set has enough questions.

Key points

• Participation: 20 organizations submitted 45 runs (20 with 50-byte answers, 25 with 250-byte answers). Scoring used the 198 questions of the official test set.
• Top results: Cymfony's textract9908 had the highest MRR (.660) among 50-byte runs, missing 54 questions. SMUNLP2 from Southern Methodist U. led the 250-byte runs with .646, missing 44 questions.
• Length comparison: every organization that submitted both lengths scored higher with its 250-byte run. AT&T's passage-based runs showed that sentence retrieval works for long answers but not for short ones.
• Judge stability: rankings of 41 runs under single-judge qrels had a mean Kendall correlation of about .96, both with each other and with the adjudicated qrels. Adjudicated scores always fell between the minimum and maximum single-judge scores.
• Next steps: the authors suggest at least 100 questions per test set. TREC-9 was planned with 500 questions and a new "exact answer" condition.

Interesting ideas

• I1. Calibrate task difficulty: the task was deliberately narrowed so systems would neither all succeed nor all fail. This principle is useful when designing any shared task or benchmark, since a task at either extreme tells you nothing about which methods are better.
• I2. Judge answers in document context: a response was credited only if it was supported by the document it came from. This rewards correct extraction even from a document that is itself wrong, and it separates system quality from source quality.
• I3. Penalize ambiguous answers: an answer string that contained several candidates of the right type, without singling one out, was marked wrong. This discourages gaming length limits by padding a response with many possible answers.
• I4. Measure ranking stability, not absolute scores: the authors compared system rankings across many simulated single-judge qrels using Kendall correlation. This is a cheap, reusable way to check whether a benchmark's annotations are reliable enough for comparing systems, even when annotators disagree.
• I5. Report the limits of the metric: the authors noted MRR's weaknesses. Per-question scores take only six values, multiple correct answers earn no extra credit, and systems cannot say "I don't know." Listing such limitations openly helps later benchmark designers decide what to fix.