Elamur Research⌕
☰
home / papers / 2007 / Using Semantic Roles to Improve Question Answering

Using Semantic Roles to Improve Question Answering

Summary

The paper asks whether FrameNet-style semantic roles can help answer extraction in open-domain factoid question answering. Earlier attempts had given weak or negative results, mostly because shallow semantic parsers handle questions poorly and FrameNet has limited coverage. The authors build semantic structures (a predicate plus scored role assignments) for questions and candidate answer sentences. They use only a dependency parser and the FrameNet database. Role labeling compares dependency relation paths in raw text with paths found in FrameNet's annotated sentences, and is solved as a globally optimal assignment in a bipartite graph. Answer extraction then becomes graph matching between the question structure and each candidate's structure. On TREC02–TREC05, the model (SemMatch) clearly outperforms a syntax-only matcher and a pipeline built on the Shalmaneser parser. Combined with the syntactic system, it beats syntax alone on the full datasets. The study also measures how often FrameNet actually applies to TREC questions, which is only about 35% of them. This shows that semantic roles help, but should complement syntax-based methods rather than replace them.

Key points

  • Two-part model. Model I generates semantic structures by matching dependency paths against FrameNet annotations, framed as bipartite-graph optimization. Model II scores question–candidate pairs by graph matching.
  • Coverage analysis. Failure cases are split into NoFrame, NoAnnot and NoMatch, with NoMatch (question and answer evoke different frames) the most common. Only about 35% of TREC questions are amenable to a FrameNet analysis.
  • Results on that ~35% subset (rank-1 accuracy):
    • SemMatch scores 53.29 / 49.11 / 54.67 / 59.65 on TREC02–05.
    • SynMatch scores 35.53 / 33.04 / 40.00 / 36.84.
    • SemParse (Shalmaneser) scores only 8.92–17.33.
  • Results on the full datasets:
    • +SemMatch (FrameNet model with SynMatch fallback) reaches 38.96 / 35.53 / 42.36 / 41.76.
    • SynMatch alone reaches 32.88 / 30.70 / 35.95 / 34.38.
    • +SemParse drops to 23.68–28.57.
    • All of these differences are significant at p < 0.01 (χ² test).
  • Why Shalmaneser struggled. It labels questions poorly, even when they are rewritten as declaratives, and it favors precision over recall.

Interesting ideas

  • I1. Project roles through dependency paths. Unlabeled text gets role labels by comparing dependency paths with those in an annotated resource, which avoids heavy feature engineering. This could carry over to lower-resource languages where a UD parser exists but a trained role labeler does not.
  • I2. Exact global assignment instead of greedy labeling. Treating role labeling as an optimization over a bipartite graph yields an exact global solution. Scored, soft assignments also tolerate gaps in resource coverage better than strict matching.
  • I3. Skip early sense disambiguation. Instead of choosing a frame for a polysemous predicate up front, the system runs role assignment for every frame the predicate can evoke. Deferring the decision to the final matching step avoids errors that would otherwise cascade.
  • I4. Fallback ensemble. The system tries the precise but narrow semantic module first and passes the question to a broad syntactic matcher when no answer is found. This simple combination suits any pairing of a high-precision, low-coverage component with a robust baseline.
  • I5. Coverage diagnosis as part of evaluation. Breaking failures into missing frame, missing annotation, and representation mismatch shows how far a resource can help at all. That kind of analysis is worth running before investing in a resource-dependent method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling

Cited by

150 papers of our library cite it (of the 140884 whose bibliographies we read); 24 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shen2007using,
  title = {{Using Semantic Roles to Improve Question Answering}},
  author = {Shen, Dan and Lapata, Mirella},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1002/}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Brill, S. Dumais, M. Banko. 2002. An analysis of the askMSR question-answering system. In Proceedings of the EMNLP, 257–264, Philadelphia, PA. find
  2. X. Carreras, L. Màrquez, eds. 2005. Proceedings of the CoNLL shared task: Semantic role labelling, 2005. find
  3. T. Cormen, C. Leiserson, R. Rivest. 1990. Introduction to Algorithms. MIT Press. find
  4. H. Cui, R. X. Sun, K. Y. Li, M. Y. Kan, T. S. Chua. 2005. Question answering passage retrieval using dependency relations. In Proceedings of the ACM SIGIR, 400–407. ACM Press. find
  5. T. Eiter, H. Mannila. 1997. Distance measures for point sets and their computation. Acta Informatica, 34(2):109–133. find
  6. K. Erk, S. Padó. 2006. Shalmaneser - a flexible toolbox for semantic role assignment. In Proceedings of the LREC, 527–532, Genoa, Italy. find
  7. C. Fellbaum, ed. 1998. WordNet. An Electronic Lexical Database. MIT Press, Cambridge/Mass. find
  8. C. J. Fillmore, C. R. Johnson, M. R. Petruck. 2003. Background to FrameNet. International Journal of Lexicography, 16:235–250. find
  9. D. Gildea, D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245– 288. find our card
  10. T. Grenager, C. D. Manning. 2006. Unsupervised discovery of a statistical verb lexicon. In Proceedings of the EMNLP, 1–8, Sydney, Australia. find
  11. R. Jonker, A. Volgenant. 1987. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38:325–340. find
  12. M. Kaisser. 2006. Web question answering by exploiting wide-coverage lexical resources. In Proceedings of the 11th ESSLLI Student Session, 203–213. find
  13. J. Leidner, J. Bos, T. Dalmas, J. Curran, S. Clark, C. Bannard, B. Webber, M. Steedman. 2004. The qed open-domain answer retrieval system for TREC 2003. In Proceedings of the TREC, 595–599. find
  14. C. Leslie, E. Eskin, W. S. Noble. 2002. The spectrum kernel: a string kernel for SVM protein classification. In Proceedings of the Pacific Biocomputing Symposium, 564–575. find
  15. B. Levin. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. find
  16. X. Li, D. Roth. 2002. Learning question classifiers. In Proceedings of the 19th COLING, 556–562, Taipei, Taiwan. find our card
  17. D. K. Lin. 1994. PRINCIPAR–an efficient, broad-coverage, principle-based parser. In Proceedings of the 15th COLING, 482–488. find
  18. D. Moldovan, C. Clark, S. Harabagiu, S. Maiorano. 2003. COGEX: A logic prover for question answering. In Proceedings of the HLT/NAACL, 87–93, Edmonton, Canada. find
  19. S. Narayanan, S. Harabagiu. 2004. Question answering based on semantic structures. In Proceedings of the 19th COLING, 184–191. find
  20. S. Padó, M. Lapata. 2006. Optimal constituent alignment with edge covers for semantic projection. In Proceedings of the COLING/ACL, 1161–1168. find
  21. M. Palmer, D. Gildea, P. Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  22. D. Paranjpe, G. Ramakrishnan, S. Srinivasa. 2003. Passage scoring for question answering via bayesian inference on lexical relations. In Proceedings of the TREC, 305–210. find
  23. S. Pradhan, W. Ward, K. Hacioglu, J. Martin, D. Jurafsky. 2004. Shallow semantic parsing using support vector machines. In Proceedings of the HLT/NAACL, 141– 144, Boston, MA. find our card
  24. D. Shen, D. Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction. In Proceedings of the COLING/ACL, 889–896. find
  25. R. X. Sun, J. J. Jiang, Y. F. Tan, H. Cui, T. S. Chua, M. Y. Kan. 2005. Using syntactic and semantic relation analysis in question answering. In Proceedings of the TREC. find
  26. B. Taskar, S. Lacoste-Julien, D. Klein. 2005. A discriminative matching approach to word alignment. In Proceedings of the HLT/EMNLP, 73–80, Vancouver, BC. find our card
  27. M. Wu, M. Y. Duan, S. Shaikh, S. Small, T. Strzalkowski. 2005. University at albany’s ilqua in trec 2005. In Proceedings of the TREC, 77–83. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2007-semantic-roles-improve-question-answering/