Elamur Research⌕
☰
home / papers / 2009 / A Comparative Study on Generalization of Semantic Roles in FrameNet

A Comparative Study on Generalization of Semantic Roles in FrameNet

Summary

FrameNet defines semantic roles separately for each frame, so many roles have very few annotated examples. This sparsity hurts machine-learned semantic role labeling. Matsubayashi, Okazaki and Tsujii study how to share training evidence across frames, focusing on the role classification step: assigning the right role to an already identified phrase. They compare four ways of grouping roles:

  • the FrameNet role-to-role hierarchy (seven relation types, including Inheritance and Using),
  • role descriptors, meaning the human-readable role names that recur across frames,
  • semantic types of the filler phrases,
  • VerbNet thematic roles obtained through SemLink.

They do not replace the original labels with coarser classes. Instead, they build feature functions that pair each role group with the observed features, so the training data decides how much weight each criterion gets. Combining all groups cuts classification errors by 19.16% and raises macro-averaged F1 by 7.42%, with clear gains on rarely seen roles. The simple descriptor grouping works best, and the hierarchy is less useful than the authors expected. The approach is not tied to FrameNet and could carry over to other SRL resources such as PropBank.

Key points

  • Overall result with all role groups: 19.16% error reduction and a 7.42% macro-averaged F1 gain over the baseline. The baseline's 89.00% micro-F1 is roughly on par with the 88.93% reported by Bejan and Hathaway (2007) for SemEval-2007.
  • Applying generalization at the feature level beat the conventional approach of swapping original role labels for generalized ones.
  • Role descriptors gave the largest benefit, especially for peripheral roles (micro-F1 rose from 78.46 to 83.51) and extra-thematic roles (from 71.87 to 74.82). Core roles were already easy, with a baseline F1 of 91.93.
  • Within the FrameNet hierarchy, Inheritance and Using relations helped slightly more than the other relation types. Grouping over all ancestors worked better than grouping over parents only, and using every relation type gave the best result.
  • SemLink covers only 1,726 FrameNet roles, about 37.61% of the roles that occur in the corpus, which limits the VerbNet-based groups. An analysis of the top-weighted features suggests that descriptor and semantic-type groups mainly help adjunct-like roles, while hierarchy groups mainly help core roles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{matsubayashi2009comparative,
  title = {{A Comparative Study on Generalization of Semantic Roles in FrameNet}},
  author = {Matsubayashi, Yuichiroh and Okazaki, Naoaki and Tsujii, Jun'ichi},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  doi = {10.3115/1687878.1687883},
  url = {https://doi.org/10.3115/1687878.1687883}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of Coling-ACL 1998, pages 86–90. find our card
  2. Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. Semeval-2007 task 19: Frame semantic struc-There were two participants that performed whole SRL in SemEval-2007. find
  3. Bejan and Hathaway (2007) evaluated role classification accuracy separately for the training data. ture extraction. In Proceedings of SemEval-2007, pages 99–104. find
  4. Ulrike Baldewein, Katrin Erk, Sebastian Padó, and Detlef Prescher. 2004. Semantic role labeling with similarity based generalization using EM-based clustering. In Proceedings of Senseval-3, pages 64– 68. find
  5. Cosmin Adrian Bejan and Chris Hathaway. 2007. UTD-SRL: A Pipeline Architecture for Extracting Frame Semantic Structures. In Proceedings of SemEval-2007, pages 460–463. Association for Computational Linguistics. find our card
  6. X. Chang and Q. Zheng. 2008. Knowledge Element Extraction for Knowledge-Based Learning Resources Organization. Lecture Notes in Computer Science, 4823:102–113. find
  7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 173–180. find our card
  8. Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proceedings of EMNLP-2006, pages 594–602. find
  9. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  10. Ana-Maria Giuglea and Alessandro Moschitti. 2006. Semantic role labeling via FrameNet, VerbNet and PropBank. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the ACL, pages 929–936. find
  11. Andrew Gordon and Reid Swanson. 2007. Generalizing semantic role annotations across syntactically similar verbs. In Proceedings of ACL-2007, pages 192–199. find
  12. Edward Loper, Szu-ting Yi, and Martha Palmer. 2007. Combining lexical resources: Mapping between propbank and verbnet. In Proceedings of the 7th International Workshop on Computational Semantics, pages 118–128. find
  13. Lluı́s Màrquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: an introduction to the special issue. Computational linguistics, 34(2):145–159. find
  14. Alessandro Moschitti, Ana-Maria Giuglea, Bonaventura Coppola, and Roberto Basili. 2005. Hierarchical semantic role labeling. In Proceedings of CoNLL-2005, pages 201–204. find
  15. Alessandro Moschitti, Silvia Quarteroni, Roberto Basili, and Suresh Manandhar. 2007. Exploiting syntactic and shallow semantic kernels for question answer classification. In Proceedings of ACL-07, pages 776–783. find
  16. Srini Narayanan and Sanda Harabagiu. 2004. Question answering based on semantic structures. In Proceedings of Coling-2004, pages 693–701. find
  17. Jorge Nocedal. 1980. Updating quasi-newton matrices with limited storage. Mathematics of Computation, 35(151):773–782. find
  18. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  19. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of EMNLP-CoNLL 2007, pages 12–21. find
  20. Lei Shi and Rada Mihalcea. 2005. Putting Pieces Together: Combining FrameNet, VerbNet and Word-Net for Robust Semantic Parsing. In Proceedings of CICLing-2005, pages 100–111. find
  21. Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of ACL-2003, pages 8–15. find
  22. Szu-ting Yi, Edward Loper, and Martha Palmer. 2007. Can semantic roles generalize across genres? In Proceedings of HLT-NAACL 2007, pages 548–555. find
  23. Beñat Zapirain, Eneko Agirre, and Lluı́s Màrquez. 2008. Robustness and generalization of role sets: PropBank vs. VerbNet. In Proceedings of ACL-08: HLT, pages 550–558. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/matsubayashi-2009-comparative-study-generalization-semantic-roles/