📄 Learning surface text patterns for a Question Answering System

Summary

Open-domain QA systems usually depend on heavy external resources such as named entity taggers, WordNet, parsers and ontologies. Yet the best system at TREC-10 relied mainly on a large inventory of surface patterns. This paper tests how far such patterns can go when they are learned automatically rather than written by hand.

For each question type, the authors start from about ten seed question–answer pairs (e.g. Mozart / 1756) and query AltaVista with both terms. They keep only sentences that mention both terms and run them through a suffix tree to enumerate every repeated substring. The surviving strings become regular expressions with <NAME> and <ANSWER> slots. A second retrieval round, using only the question term, estimates each pattern's precision by cross-checking it on the other seed examples. At answer time, matching patterns extract candidates, which are ranked by pattern precision.

On six TREC-10 question types, scored with MRR, web-sourced answers clearly beat answers drawn from the TREC corpus. The method's near-absence of language-specific tooling makes it attractive for rapid multilingual QA.

Key points

• Question types and data. Six types were taken from the Webclopedia QA Typology: BIRTHDATE, LOCATION, INVENTOR, DISCOVERER, DEFINITION and WHY-FAMOUS. Each seed query to AltaVista retrieves the top 1000 documents.

• Precision estimate. Precision is computed as P = Ca / Co. Co counts sentences where the pattern matches with any word in the answer slot. Ca counts sentences where the slot holds the correct answer. Only patterns that match more than 5 examples are retained.

• Example precisions. For BIRTHDATE, the pattern "<NAME> ( <ANSWER> - )" scores 1.0, while the looser "<NAME> was born in <ANSWER>" scores 0.6. The system returns the top 5 answers after removing duplicates.

• MRR on the TREC corpus:

  • BIRTHYEAR: 0.48 (8 questions)
  • INVENTOR: 0.17 (6 questions)
  • DISCOVERER: 0.13 (4 questions)
  • DEFINITION: 0.34 (102 questions)
  • WHY-FAMOUS: 0.33 (3 questions)
  • LOCATION: 0.75 (16 questions)

  Web retrieval performed notably better. The authors explain this by the web's redundancy: a short pattern usually matches somewhere, without needing long-distance dependencies.

• Reported weaknesses:

  • no semantic or POS typing of the answer slot;
  • a single anchor term per pattern;
  • a fixed 50-byte answer window;
  • no case sensitivity;
  • name and date variants must be listed manually.

Interesting ideas

• I1. Suffix trees for pattern mining. Suffix trees find repeated substrings of any length in linear time. Because overlapping variants are stored separately, a specific pattern and its more general substring each get their own precision. This is useful for any template-mining task where the right pattern length is unknown.
• I2. Cross-example precision checks. Patterns learned from one seed pair are scored on documents retrieved for different seeds of the same type. This filters out artifacts caused by mirrored or duplicated web pages, a cheap safeguard for any noisy web-bootstrapped resource.
• I3. Precision as a simple likelihood estimate. Comparing "slot matches anything" against "slot matches the gold answer" gives a quick reliability score for extraction rules. Those scores can then rank candidate answers directly, with no extra model.
• I4. Seed-based bootstrapping instead of annotation. A handful of seed pairs plus a search engine replaces hand-tagged corpora. The authors stress that this makes porting to a new language fast, which suits lower-resource settings such as Ukrainian.
• I5. Suggested extensions. The authors propose bounding the answer slot by 1–2 syntactic chunks instead of raw bytes. They also suggest requiring several question terms as anchors, which would prune many false candidates. Both are simple constraints worth adding to pattern-based extractors.