Elamur Research⌕
☰
home / papers / 2006 / Methods for Using Textual Entailment in Open-Domain Question Answering

Methods for Using Textual Entailment in Open-Domain Question Answering

Paper · 2006 · Sanda Harabagiu and Andrew Hickl

Summary

Harabagiu and Hickl ask whether a system built to recognize textual entailment (TE) can make open-domain question answering (Q/A) more accurate. The idea comes from formal semantics, where a correct answer is seen as something the question logically entails. The authors pair questions with candidate answers or passages and send these pairs to a supervised TE classifier, LCC's Groundhog-style pipeline. That pipeline has three stages: preprocessing, Maximum Entropy alignment with paraphrase acquisition, and a decision-tree classifier. They insert TE at three points in a standard question processing → passage retrieval → answer processing architecture:

  • Method 1 filters and re-ranks final answers.
  • Method 2 re-ranks retrieved passages.
  • Method 3 checks entailment between the user question and questions generated automatically from passages (AutoQUAB).

A Hybrid combination is also tested. With TE, accuracy on questions with a known expected answer type rose from 32.0% to 53.9% (Hybrid). On questions without one, it rose from 30.6% to at most 42.7%. The work matters because it treats answer validation as a semantic inference problem rather than a matter of relevance alone.

Key points

  • The baseline Q/A system scored 32.0% accuracy (MRR 0.3001) when the expected answer type (EAT) was known and 30.6% when it was not.

  • Results with a known EAT:

    • Method 1: 44.1%
    • Method 2: 52.4% (MRR 0.5558)
    • Method 3: 41.5%
    • Hybrid: 53.9% (MRR 0.5640)

  • Without an EAT, Method 2 scored best at 42.7%, slightly ahead of the Hybrid at 41.9%.

  • In Methods 2 and 3, the top 50 passages or candidate answers were paired with the question and sent to the TE system.

  • The TE system's preprocessing covers:

    • parsing and semantic dependencies
    • NER
    • temporal and spatial normalization
    • coreference
    • polarity, tense and modality annotation

    Alignment uses a MaxEnt classifier, and the final judgment comes from decision trees.

Interesting ideas

  • I1. Reuse an entailment model as an answer validator. Swapping RTE text–hypothesis pairs for question–answer pairs gives a general-purpose verifier for retrieval pipelines, with no need to build a dedicated answer-checking model.
  • I2. Apply TE before answer extraction, not only after it. Ranking passages by entailment confidence reduces the load on the expensive extraction step, and in this paper it was also the most effective single method. This is a cheap pre-filter worth trying in modern retrieval-augmented setups.
  • I3. Compare questions with questions. Generating questions from candidate passages and checking whether the user's question entails them sidesteps the mismatch in form between a question and a declarative answer.
  • I4. Feed confirmed pairs back as training data. Question–answer pairs that pass the entailment check are stored and used to further train the question generator, a simple self-training loop.
  • I5. Report results separately by question condition. Splitting results into known versus unknown expected answer type shows where the gains actually come from. Here the gains were much larger when the EAT was known, which points to answer-type-aware TE features as the next step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Language ComputerRichardson · United States2 works · 781 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works

🧭 Topics 2 Reasoning · 510 works Question answering · 242 works

Cite

Cite as Sanda Harabagiu and Andrew Hickl. 2006. Methods for Using Textual Entailment in Open-Domain Question Answering. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1220175.1220289

@inproceedings{harabagiu2006methods,
  title = {{Methods for Using Textual Entailment in Open-Domain Question Answering}},
  author = {Sanda Harabagiu and Andrew Hickl},
  booktitle = {Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
  doi = {10.3115/1220175.1220289},
  url = {https://doi.org/10.3115/1220175.1220289}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In HLT-NAACL. our card
  2. John Burger and Lisa Ferro. 2005. Generating an Entailment Corpus from News Headlines. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 49–54. find
  3. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognizing Textual Entailment Challenge. In Proceedings of the PASCAL Challenges Workshop. find
  4. Abdessamad Echihabi and Daniel Marcu. 2003. A noisy-channel approach to question answering. In Proceedings of the 41st Meeting of the Association for Computational Linguistics. find
  5. Oren Glickman and Ido Dagan. 2005. A Probabilistic Setting and Lexical Co-occurrence Model for Textual Entailment. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, USA. find
  6. Jeroen Groenendijk. 1999. The logic of interrogation: Classical version. In Proceedings of the Ninth Semantics and Linguistics Theory Conference (SALT IX), Ithaca, NY. find
  7. Sanda Harabagiu, Dan Moldovan, Marius Pasca, Rada Mihalcea, Mihai Surdeanu, Razvan Bunsecu, Roxana Girju, Vasile Rus, and Paul Morarescu. 2001. The Role of Lexico-Semantic Feedback in Open-Domain Textual Question-Answering. In Proceedings of the 39th Meeting of the Association for Computational Ling… find
  8. S. Harabagiu, D. Moldovan, C. Clark, M. Bowden, A. Hickl, and P. Wang. 2005a. Employing Two Question Answering Systems in TREC 2005. In Proceedings of the Fourteenth Text REtrieval Conference. find
  9. Sanda Harabagiu, Andrew Hickl, John Lehmann, and Dan Moldovan. 2005b. Experiments with Interactive Question-Answering. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). find
  10. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing Textual Entailment with LCC’s Groundhog System. In Proceedings of the Second PASCAL Challenges Workshop. find
  11. David Lewis. 1988. Relevant Implication. Theoria, 54(3):161–174. find
  12. Bernardo Magnini, Matteo Negri, Roberto Prevete, and Hristo Tanev. 2002. Is it the right answer? exploiting web redundancy for answer validation. In Proceedings of the Fortieth Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA. find
  13. Dan Moldovan, Marius Pasca, Sanda Harabagiu, and Mihai Surdeanu. 2002. Performance Issues and Error Analysis in an Open-Domain Question Answering System. In Proceedings of the 4Oth Meeting of the Association for Computational Linguistics. find
  14. Dan Moldovan, Christine Clark, Sanda Harabagiu, and Steve Maiorano. 2003. COGEX: A Logic Prover for Question Answering. In Proceedings of HLT/NAACL-2003. find
  15. T. Pedersen, S. Patwardhan, and J. Michelizzi. 2004. WordNet::Similarity - Measuring the Relatedness of Concepts. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04), San Jose, CA. find
  16. John Prager, Jennifer Chu-Carroll, and Krzysztof Czuba. 2004. Question answering using constraint satisfaction: Qa-by-dossier-with-contraints. In Proceedings of the ACL-2004, pages 574–581, Barcelona, Spain, July. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/harabagiu-2006-methods-textual-entailment-open-domain/