Elamur Research⌕
☰
home / papers / 2008 / Special Issue Introduction: Semantic Role Labeling: An Introduction to the Special Issue

Special Issue Introduction: Semantic Role Labeling: An Introduction to the Special Issue

Paper · 2008 · Lluís Màrquez, Xavier Carreras, Kenneth C. Litkowski et al.

Summary

This article opens a Computational Linguistics special issue on semantic role labeling (SRL), the task of finding a predicate's participants and circumstances in a sentence and assigning each a role such as Agent, Recipient or Temporal. The authors first survey the linguistic background: Fillmore's case and frame semantics, Dowty's proto-roles, and Levin-style verb classes built on syntactic alternations. They then show how these theories shaped annotated resources such as FrameNet, PropBank, NomBank and VerbNet, and how those resources made statistical, supervised SRL possible. The article outlines typical system pipelines and reviews results from CoNLL-2004/2005, Senseval-3 and SemEval-2007. The main finding is that English systems reach F1 in the low 80s on in-domain data but degrade sharply on new genres. They also depend heavily on parser quality and on costly annotation. The authors argue that the field should explore unsupervised and lightly supervised learning, connect roles to other semantic knowledge, and rely less on narrow benchmark gains.

Key points

  • FrameNet combines a lexicon of situation frames with an annotated corpus of over 141,000 sentences, taken mostly from the British National Corpus plus some additional texts. Gildea and Jurafsky (2002) used it to build the first statistical SRL system.
  • In CoNLL-2005, systems trained on WSJ scored about 80% F1 on WSJ but below 70% on Brown corpus sections. Pradhan, Ward and Martin trace the loss mainly to role assignment, not to finding argument boundaries.
  • SemEval-2007 brought the first non-English SRL evaluation, for Spanish and Catalan on CESS-ECE (about 100K tokens per language, gold syntax). The best scores were about 86% for predicate disambiguation and about 83% for argument labeling.
  • The SemEval-2007 preposition task used over 25,000 FrameNet sentences covering 34 frequent English prepositions. Nearly 80% of the prepositional phrases turned out to be core frame elements.
  • Many SRL errors come from wrong syntactic constituents. Top CoNLL systems therefore combined outputs from several parsers to gain robustness.

Interesting ideas

  • I1. Feeding role labeling with several alternative parses lets the system recover when one parse has a wrong constituent.
  • I2. Errors should be split into argument-boundary errors and role-choice errors, because they behave differently under domain shift.
  • I3. CESS-ECE gives each argument two labels: a verb-specific one in PropBank style and a general VerbNet-like thematic role.
  • I4. A preposition sometimes only marks a role the verb assigns (the "to" in giving to Mary) and sometimes acts as a predicate with its own roles.
  • I5. Systems label core arguments more reliably than peripheral ones such as Manner, Time and Place.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver frame data could be built by checking roles across several witness parsers, keeping only role assignments that survive in most trees.
  • I2 (learn): Frame evaluation could report span detection and role labeling separately, showing whether a drop on tales or school texts comes from the tree or from the role lexicon.
  • I3 (frames): The prag crate could store both a verb-local argument slot and a VerbNet thematic role, so reasoning can generalize while exact verb senses stay recoverable.
  • I4 (frames): Frame building could decide per preposition whether to fold it into the verb's role or open a separate relation, avoiding duplicate or missing participants.
  • I5 (answer): Readers could attach confidence by role type and treat adjunct-based answers (when, how) as needing extra checks against the world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓University of TorontoToronto · Canada96 works · 142,007 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations🎓Universitat Politècnica de CatalunyaBarcelona · Spain31 works · 15,636 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semantic role labeling · 145 works

Cite

Cite as Lluís Màrquez, Xavier Carreras, Kenneth C. Litkowski and Suzanne Stevenson. 2008. Special Issue Introduction: Semantic Role Labeling: An Introduction to the Special Issue. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2008.34.2.145

@article{marquez2008special,
  title = {{Special Issue Introduction: Semantic Role Labeling: An Introduction to the Special Issue}},
  author = {Lluís Màrquez, Xavier Carreras, Kenneth C. Litkowski and Suzanne Stevenson},
  journal = {Computational Linguistics},
  year = {2008},
  doi = {10.1162/coli.2008.34.2.145},
  url = {https://doi.org/10.1162/coli.2008.34.2.145}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baker, C., M. Ellsworth, and K. Erk. 2007. SemEval-2007 Task 19: Frame semantic structure extraction. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 99–104, Prague, Czech Republic.
  2. Boas, H. C. 2002. Bilingual framenet dictionaries for machine translation. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC), pages 1364–1371, Las Palmas de Gran Canaria, Spain.
  3. Briscoe, T. and J. Carroll. 1997. Automatic extraction of subcategorization from corpora. In Proceedings of the 5th ACL Conference on Applied Natural Language Processing (ANLP), pages 356–363, Washington, DC. ↗
  4. Carreras, X. and L. Màrquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004), pages 89–97, Boston, MA. find
  5. Carreras, X. and L. Màrquez. 2005. Introduction to the CoNLL-2005 Shared Task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, MI.
  6. Carreras, X., L. Màrquez, and G. Chrupała. 2004. Hierarchical recognition of propositional arguments with perceptrons. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004), pages 106–109, Boston, MA. find
  7. Cohn, T. and P. Blunsom. 2005. Semantic role labelling with tree conditional random fields. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 169–172, Ann Arbor, MI. find
  8. Copestake, A. and D. Flickinger. 2000. An open-source grammar development environment and broad-coverage English grammar using HPSG. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC), pages 591–600, Athens, Greece. find
  9. Dowty, D. 1991. Thematic proto-roles and argument selection. Language, 67:547–619. ↗
  10. Erk, K. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 216–223, Prague, Czech Republic. find
  11. Fillmore, C. 1968. The case for case. In E. Bach and R. T. Harms, editors, Universals in Linguistic Theory. Holt, Rinehart & Winston, New York, pages 1–88. find
  12. Fillmore, C. J. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language and Speech, 280:20–32. ↗
  13. Fillmore, C. J., J. Ruppenhofer, and C. F. Baker. 2004. Framenet and representing the link between semantic and syntactic relations. In Churen Huang and Winfried Lenders, editors, Frontiers in Linguistics, volume I of Language and Linguistics Monograph Series B. Institute of Linguistics, Academia Sinica, Taipei, pages 19–59. find
  14. Gildea, D. and D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288.
  15. Grimshaw, J. 1990. Argument Structure. MIT Press, Cambridge, MA. find
  16. Habash, N., B. J. Dorr, and D. Traum. 2003. Hybrid natural language generation from lexical conceptual structures. Machine Translation, 18(2):81–128. ↗
  17. Hirst, G. 1987. Semantic Interpretation and the Resolution of Ambiguity. Cambridge University Press, Cambridge. ↗
  18. Jackendoff, R. 1990. Semantic Structures. MIT Press, Cambridge, MA. find
  19. Johansson, R. and P. Nugues. 2007. LTH: Semantic structure extraction using nonprojective dependency trees. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 227–230, Prague, Czech Republic. find
  20. Kipper, K., H. T. Dang, and M. Palmer. 2000. Class-based construction of a verb lexicon. In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), Austin, TX. find
  21. Koomen, P., V. Punyakanok, D. Roth, and W. Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 181–184, Ann Arbor, MI. find
  22. Levin, B. 1993. English Verb Classes and Alternations: A Preliminary Investigation. The University of Chicago Press, Chicago, IL. find
  23. Levin, B. and M. Rappaport Hovav. 1998. Building verb meanings. In M. Butt and W. Geuder, editors, The Projection of Arguments: Lexical and Compositional Factors. CSLI Publications, Stanford, CA, pages 97–134. find
  24. Levin, B. and M. Rappaport Hovav. 2005. Argument Realization. Cambridge University Press, Cambridge. find
  25. Litkowski, K. C. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Proceedings of the 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text (Senseval-3), pages 9–12, Barcelona, Spain. find
  26. Litkowski, K. C. and O. Hargraves. 2005. The preposition project. In Proceedings of the ACL-SIGSEM Workshop on the Linguistic Dimensions of Prepositions and their Use in Computational Linguistic Formalisms and Applications, pages 171–179, Colchester, UK. find
  27. Litkowski, K. C. and O. Hargraves. 2007. SemEval-2007 Task 06: Word-sense disambiguation of prepositions. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 24–29, Prague, Czech Republic. find
  28. Loper, E., S. Yi, and M. Palmer. 2007. Combining lexical resources: Mapping between PropBank and VerbNet. In Proceedings of the 7th International Workshop on Computational Semantics, pages 118–128, Tilburg, The Netherlands. find
  29. Màrquez, L., P. R. Comas, J. Giménez, and N. Català. 2005. Semantic role labeling as sequential tagging. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 193–196, Ann Arbor, MI. find
  30. Màrquez, L., L. Villarejo, M.A. Martı́, and M. Taulé. 2007. SemEval-2007 Task 09: Multilevel semantic annotation of Catalan and Spanish. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 42–47, Prague, Czech Republic. find
  31. Melli, G., Y. Wang, Y. Liu, M. M. Kashani, Z. Shi, B. Gu, A. Sarkar, and F. Popowich. 2005. Description of SQUASH, the SFU question answering summary handler for the DUC-2005 Summarization Task. In Proceedings of the HLT/EMNLP Document Understanding Workshop (DUC), Vancouver, Canada, available at http://duc.nist.gov/pubs/2005papers/simonfraseru.sarkar.pdf. ↗
  32. Merlo, P. and S. Stevenson. 2001. Automatic verb classification based on statistical distributions of argument structure. Computational Linguistics, 27(3):373–408. ↗
  33. Meyers, A., R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The NomBank Project: An interim report. In Proceedings of the HLT-NAACL 2004 Workshop: Frontiers in Corpus Annotation, pages 24–31, Boston, MA.
  34. Moldovan, D., A. Badulescu, M. Tatu, D. Antohe, and R. Girju. 2004. Models for the semantic classification of noun phrases. In Proceedings of the HLT-NAACL 2004 Workshop on Computational Lexical Semantics, pages 60–67, Boston, MA. find
  35. Musillo, G. and P. Merlo. 2006. Accurate parsing of the proposition bank. In Proceedings of the Human Language Technology Conference of the NAACL, pages 101–104, New York, NY. find
  36. Narayanan, S. and S. Harabagiu. 2004. Question answering based on semantic structures. In Proceedings of the 20th International Conference on Computational Linguistics (COLING), pages 693–701, Geneva, Switzerland. find
  37. O’Hara, T. and J. Wiebe. 2003. Preposition semantic classification via Penn Treebank and FrameNet. In Proceedings of the Seventh Conference on Computational Natural Language Learning (CoNLL-2003), pages 79–86, Edmonton, Canada. find
  38. Palmer, M., D. Gildea, and P. Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–105. ↗
  39. Pradhan, S., K. Hacioglu, V. Krugler, W. Ward, J. Martin, and D. Jurafsky. 2005a. Support vector learning for semantic argument classification. Machine Learning, 60(1):11–39. ↗
  40. Pradhan, S., K. Hacioglu, W. Ward, J. H. Martin, and D. Jurafsky. 2005b. Semantic role chunking combining complementary syntactic views. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 217–220, Ann Arbor, MI. find
  41. Pradhan, S., E. Loper, D. Dligach, and M. Palmer. 2007. SemEval-2007 Task 17: English lexical sample, SRL and all words. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92, Prague, Czech Republic.
  42. Pustejovsky, J. 1995. The Generative Lexicon. MIT Press, Cambridge, MA. ↗
  43. Rosario, B. and M. Hearst. 2004. Classifying semantic relations in bioscience text. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 430–437, Barcelona, Spain. find
  44. Schulte im Walde, S. 2006. Experiments on the automatic induction of German semantic verb classes. Computational Linguistics, 32(2):159–194. ↗
  45. Shi, L. and R. Mihalcea. 2005. Putting pieces together: Combining FrameNet, VerbNet and WordNet for robust semantic parsing. In Computational Linguistics and Intelligent Text Processing; Sixth International Conference, CICLing 2005, Proceedings, LNCS, vol 3406, pages 100–111, Mexico City, Mexico. find
  46. Surdeanu, M., S. Harabagiu, J. Williams, and P. Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 8–15, Sapporo, Japan.
  47. Surdeanu, M., L. Màrquez, X. Carreras, and P. R. Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelligence Research (JAIR), 29:105–151. ↗
  48. Swier, R. and S. Stevenson. 2004. Unsupervised semantic role labelling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 95–102, Barcelona, Spain. find
  49. Swier, R. and S. Stevenson. 2005. Exploiting a verb lexicon in automatic semantic role labelling. In Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMLNP), pages 883–890, Vancouver, B.C., Canada. find
  50. Thompson, C. A., R. Levy, and C. Manning. 2003. A generative model for semantic role labeling. In Proceedings of the 14th European Conference on Machine Learning (ECML), pages 397–408, Dubrovnik, Croatia. find
  51. Toutanova, K., A. Haghighi, and C. Manning. 2005. Joint learning improves semantic role labeling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 589–596, Ann Arbor, MI.
  52. Xue, N. and M. Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 88–94, Barcelona, Spain. find
  53. Yi, S., E. Loper, and M. Palmer. 2007. Can semantic roles generalize across corpora? In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 548–555, Rochester, NY. find
  54. Zapirain, B., E. Agirre, and L. Màrquez. 2007. UBC-UPC: Sequential SRL using selectional preferences: an approach with maximum entropy Markov models. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 354–357, Prague, Czech Republic. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marquez-2008-special-issue-introduction-semantic-role/