Elamur Research⌕
☰
home / papers / 2010 / Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet

Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet

Summary

Supervised FrameNet-style semantic role labeling (SRL) systems often report accuracies around 80%. Those figures, however, come from lemma-level evaluations on hand-picked predicates that have training data. Palmer and Sporleder examine what happens when such systems are evaluated on whole documents, where FrameNet's incomplete coverage becomes the main obstacle.

They propose a typology of five kinds of coverage gaps in FrameNet. They then measure coverage and run the Shalmaneser frame-assignment system on three full-text data sets: the FN1.3 lexicographic corpus (FNL), the FN1.3 full-text corpus (FNF), and texts annotated for SemEval 2010 Task-10.

Frame-assignment accuracy was only 37–47%. Few of these errors were genuine classifier mistakes. Most arose because FrameNet lacked the needed lemma, frame, or training examples. The authors conclude that tuning features or learning algorithms will gain little, and that research should focus on handling unseen data. As one such direction, they explore generalizing across lemmas and frames that overlap in FrameNet.

Key points

  • On average, 46.8% of targets (41–48% per data set) are "no-chance" cases, where FrameNet offers nothing for the target or the frame. The abstract calls this figure an oracle upper bound.
  • The analysis covers more than 150,000 frame-assignment targets across FNL, FNF, and SemEval 2010 Task-10. Shalmaneser's frame-assignment accuracy ranges from 37% to 47%.
  • Over 15% of lexical units in the SemEval gold annotations are absent from FrameNet. This matches Baker et al. (2007), who had to add 40 new frames for a SemEval 2007 test set containing 272 frame types.
  • Overlap is a possible remedy:
    • Lemma overlap: 8,358 of FrameNet's 8,370 lemmas share at least one frame with another lemma.
    • Frame overlap: the 502 frames that have annotated examples average 278 training instances each, ranging from 1 to 6,233 (SELF_MOTION).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{palmer2010evaluating,
  title = {{Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet}},
  author = {Palmer, Alexis and Sporleder, Caroline},
  booktitle = {Coling 2010: Posters},
  year = {2010},
  url = {https://aclanthology.org/C10-2107/}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Baker, M. Ellsworth, K. Erk. 2007. Semeval-2007 task 19: Frame semantic structure extraction. In Proceedings of SemEval-2007. find our card
  2. A. Burchardt, K. Erk, A. Frank. 2005. A WordNet Detour to FrameNet. In Proceedings of the GLDV-05 Workshop GermaNet II. find
  3. A. Burchardt, M. Pennacchiotti, S. Thater, M. Pinkal. 2009. Assessing the impact of frame semantics on textual entailment. Journal of Natural Language Engineering, Special Issue on Textual Entailment, 15(4):527–550. find
  4. D. D. Cao, D. Croce, M. Pennacchiotti, R. Basili. 2008. Combining word sense and usage for modeling frame semantics. In Proceedings of STEP-08. find our card
  5. K. Deschacht, M.-F. Moens. 2009. Semi-supervised Semantic Role Labeling Using the Latent Words Language Model. In Proceedings of EMNLP-09. find
  6. M. Ellsworth, K. Erk, P. Kingsbury, S. Padó. 2004. find
  7. PropBank, SALSA, and FrameNet: How Design Determines Product. In Proceedings LREC 2004 Workshop on Building Lexical Resources from Semantically Annotated Corpora. find
  8. K. Erk, S. Padó. 2006. Shalmaneser – a toolchain for shallow semantic parsing. In Proceedings of LREC-06. find
  9. K. Erk. 2005. Frame assignment as word sense disambiguation. In Proceedings of IWCS 6. find
  10. H. Fürstenau, M. Lapata. 2009a. Graph alignment for semi-supervised semantic role labeling. In Proceedings of EMNLP 2009. find
  11. H. Fürstenau, M. Lapata. 2009b. Semi-supervised semantic role labeling. In Proceedings of EACL 2009. find
  12. D. Gildea, D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  13. A. Gordon, R. Swanson. 2007. Generalizing semantic role annotations across syntactically similar verbs. In Proceedings of ACL 2007. find
  14. R. Johansson, P. Nugues. 2007. Using WordNet to extend FrameNet coverage. In Proceedings of the Workshop on Building Frame-semantic Resources for Scandinavian and Baltic Languages, NODAL-IDA. find
  15. M. Marcus, G. Kim, M. A. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz, B. Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In ARPA Human Language Technology Workshop. find our card
  16. D. McCarthy. 2009. Word Sense Disambiguation: An Overview. Language and Linguistics Compass, 3(2):537–558. find
  17. A. Moschitti, P. Morarescu, S. Harabagiu. 2003. Open-domain information extraction via automatic semantic labeling. In Proceedings of FLAIRS. find
  18. A. Moschitti, D. Pighin, R. Basili. 2008. Tree Kernels for Semantic Role Labeling. Computational Linguistics, 34(2). find
  19. S. Padó, M. Pennacchiotti, C. Sporleder. 2008. Semantic role assignment for event nominalisations by leveraging verbal data. In Proceedings of Coling 2008. find
  20. M. Palmer, D. Gildea, P. Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1):71–105. find our card
  21. M. Pennacchiotti, D. D. Cao, R. Basili, D. Croce, M. Roth. 2008a. Automatic induction of FrameNet lexical units. In Proceedings of EMNLP-08. find our card
  22. M. Pennacchiotti, D. D. Cao, P. Marocco, R. Basili. 2008b. Towards a Vector Space Model for FrameNet-like Resources. In Proceedings of LREC-08. find
  23. J. Ruppenhofer, M. Ellsworth, M. R. L. Petruck, C. R. Johnson, J. Scheffczyk. 2006. FrameNet II: Extended Theory and Practice. find
  24. J. Ruppenhofer, C. Sporleder, R. Morante, C. Baker, M. Palmer. 2009. SemEval-2010 Task 10: Linking Events and Their Participants in Discourse. In Proceedings of SEW-2009. find
  25. J. Ruppenhofer, M. Pinkal, J. Sunde. 2010. Generating FrameNets of various granularities. In Proceedings of LREC 2010. find
  26. D. Shen, M. Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of EMNLP-2007. find
  27. M. Surdeanu, S. Harabagiu, J. Williams, P. Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of ACL 2003. find
  28. K. Toutanova, A. Haghighi, C. D. Manning. 2008. A Global Joint Model for Semantic Role Labeling. Computational Linguistics, 34(2). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/palmer-2010-evaluating-framenet-style-semantic-parsing/