Elamur Research⌕
☰
home / papers / 2007 / Statistical Machine Translation for Query Expansion in Answer Retrieval

Statistical Machine Translation for Query Expansion in Answer Retrieval

Paper · 2007 · Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis et al.

Summary

The paper tackles the vocabulary mismatch between user questions and the answers that would satisfy them, focusing on answer retrieval from FAQ pages. The authors treat this as a query expansion problem and solve it with phrase-based SMT. They use two expansion strategies. The first is a question-to-answer "translation" model trained on roughly 10 million question-answer pairs mined from FAQ pages. The second is a full-sentence paraphraser whose phrase table is derived from bilingual phrase tables (following Bannard and Callison-Burch) and refined with discriminative training on manually created paraphrases. In both cases, new content words are taken from the n-best outputs for the whole query, so a large n-gram language model helps resolve ambiguity in context. The expanded queries are run through the tfidf answer retrieval setup of Jijkoun and de Rijke (2005). Compared with that unexpanded baseline and with local pseudo-relevance-feedback expansion (Xu and Croft), SMT-based expansion clearly raises the share of queries that get an adequate answer. This shows that MT machinery can serve as a context-aware, corpus-wide source of expansion terms.

Key points

  • Corpus construction:
    • From a 4-billion-page web subset, URL matching on "faq"/"faqs" yielded 2.6M pages (0.07% of the crawl).
    • A passive-aggressive classifier with 20 features, trained on 1,000 hand-labelled pages, reached about 90% F1 and kept 795,483 FAQ pages.
  • QA pair extraction:
    • Extraction produced 10,568,160 QA pairs, about 13.3 per FAQ page.
    • Manual checking of 100 documents (1,303 pairs) showed 98% precision and 82% recall.
  • QA translation model setup:
    • There was no discriminative weight tuning; 4,000 held-out pairs were used for manual development and testing instead.
    • The null-word alignment probability was set to 0.9 to handle the length gap between questions and answers.
    • Phrases were extracted from the intersection of alignments in both directions.
  • Evaluation design:
    • Two judges rated the top 20 results for 60 queries on a 0/1/2 scale (unsatisfactory / material / adequate).
    • Results were reported with success rates S2@n and S1,2@n rather than MRR.
  • Results (relative gains in S2 over the tfidf baseline):
    • At top 10, SMT-based expansion gained 40.7%, versus 11.1% for local expansion.
    • At top 20, the gains were 22.8% versus 14.2%.
    • On S1,2, SMT-based expansion stayed level, while local expansion dropped slightly.

Interesting ideas

  • I1. Questions and answers as two "languages": A standard parallel-corpus MT pipeline can then learn lexical bridges between them, without aiming for fluent output. The same trick could connect any paired texts with different vocabularies, such as queries and product descriptions, or colloquial and official place names.
  • I2. Expanding from n-best outputs of the whole query: Taking new terms from full-sentence translations, rather than expanding word by word, lets the language model filter out senses that do not fit the context. This is useful wherever thesaurus-style expansion introduces noise from ambiguous words.
  • I3. Bilingual phrase tables as a synonym source: Pivoting through another language turns existing bilingual phrase tables into a high-coverage paraphrase resource. This is attractive for languages with weak monolingual thesauri but available parallel data.
  • I4. Precision-first mining of web data: A small hand-labelled sample trains a light classifier, followed by cue-based field extraction with label propagation. The result is cleaner training data at some cost in recall, a good trade-off when the data feeds model training.
  • I5. Coverage-oriented evaluation with reusable judgments: - Success-rate metrics reward systems that find answers for more queries, which MRR may not do in low-recall settings. - Storing judge ratings by item ID, and reconciling disagreements between the two judges, keeps assessments consistent across systems. - The stored ratings also cut the manual labelling effort for each new system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 958 worksInformation retrieval & search · 145 works

Cite

Cite as Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal and Yi Liu. 2007. Statistical Machine Translation for Query Expansion in Answer Retrieval. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{riezler2007statistical,
  title = {{Statistical Machine Translation for Query Expansion in Answer Retrieval}},
  author = {Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal and Yi Liu},
  booktitle = {Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics},
  year = {2007},
  url = {https://aclanthology.org/P07-1059/}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eugene Agichtein, Steve Lawrence, and Luis Gravano. 2004. Learning to find answers to questions on the web. ACM Transactions on Internet Technology, 4(2):129–162. find
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of (ACL’05), Ann Arbor, MI.
  3. Adam L. Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. 2000. Bridging the lexical chasm: Statistical approaches to answer-finding. In Proceedings of SIGIR’00, Athens, Greece. find
  4. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  5. Robin B. Burke, Kristian J. Hammond, and Vladimir A. Kulyukin. 1997. Question answering from frequently-asked question files: Experiences with the FAQ finder system. AI Magazine, 18(2):57–66. find
  6. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yo-ram Singer. 2006. Online passive-agressive algorithms. Machine Learning, 7:551–585. find
  7. Pablo Ariel Duboue and Jennifer Chu-Carroll. 2006. Answering the question you wish they had asked: The impact of paraphrasing for question answering. In Proceedings of (HLT-NAACL’06), New York, NY. find
  8. Abdessamad Echihabi and Daniel Marcu. 2003. A noisy-channel approach to question answering. In Proceedings of (ACL’03), Sapporo, Japan. find
  9. Sanda Harabagiu and Finley Lacatusu. 2004. Strategies for advanced question answering. In Proceedings of the HLT-NAACL’04 Workshop on Pragmatics of Question Answering, Boston, MA. find
  10. Sanda Harabagiu, Dan Moldovan, Marius Paşca, Rada Mihalcea, Mihai Surdeanu, Răzvan Bunescu, Roxana Gı̂rju, Vasile Rus, and Paul Morărescu. 2001. The role of lexico-semantic feedback in open-domain textual question-answering. In Proceedings of (ACL’01), Toulouse, France. find
  11. Ulf Hermjakob, Abdessamad Echihabi, and Daniel Marcu. 2002. Natural language based reformulation resource and web exploitation for question answering. In Proceedings of TREC-11, Gaithersburg, MD. find
  12. Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Michael Junk, and Chin-Yew Lin. 2000. Question answering in webclopedia. In Proceedings of TREC 9, Gaithersburg, MD. find
  13. Abraham Ittycheriah, Martin Franz, and Salim Roukos. 2001. IBM’s statistical question answering system. In Proceedings of TREC 10, Gaithersburg, MD. find
  14. Valentin Jijkoun and Maarten de Rijke. 2005. Retrieving answers from frequently asked questions pages on the web. In Proceedings of the Tenth ACM Conference on Information and Knowledge Management (CIKM’05), Bremen, Germany. find
  15. Thorsten Joachims. 2003. Transductive learning via spectral graph partitioning. In Proceedings of ICML’03, Washington, DC. find
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of (HLT-NAACL’03), Edmonton, Cananda. find
  17. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question answering. Journal of Natural Language Engineering, 7(3):343–360. find
  18. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. find
  19. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of (HLT-NAACL’03), Edmonton, Cananda. find
  20. John Prager, Jennifer Chu-Carroll, and Krysztof Czuba. 2001. Use of wordnet hypernyms for answering what-is questions. In Proceedings of TREC 10, Gaithersburg, MD. find
  21. Yonggang Qiu and H. P. Frei. 1993. Concept based query expansion. In Proceedings of SIGIR’93, Pittsburgh, PA. find
  22. Dragomir R. Radev, Hong Qi, Zhiping Zheng, Sasha Blair-Goldensohn, Zhu Zhang, Weigo Fan, and John Prager. 2001. Mining the web for answers to natural language questions. In Proceedings of (CIKM’01), Atlanta, GA. find
  23. Radu Soricut and Eric Brill. 2006. Automatic question answering using the web: Beyond the factoid. Journal of Information Retrieval - Special Issue on Web Information Retrieval, 9:191–206. find
  24. Ellen M. Voorhees. 1994. lexical-semantic relations. GIR’94, Dublin, Ireland. find
  25. Query expansion using In Proceedings of SI-Jinxi Xu and W. Bruce Croft. 1996. Query expansion using local and global document analysis. In Proceedings of SIGIR’96, Zurich, Switzerland. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riezler-2007-statistical-machine-translation-query-expansion/