Elamur Research⌕
☰
home / papers / 2007 / SemEval-2007 Task 19: Frame Semantic Structure Extraction

SemEval-2007 Task 19: Frame Semantic Structure Extraction

Summary

This SemEval-2007 shared task asked systems to produce full FrameNet-style analyses of running text. Systems had to find the words that evoke semantic frames, choose the right frame (a WSD-like step), and mark the phrases filling each frame's roles (frame elements, FEs). The labels then had to fit together into a sentence-level semantic dependency graph. Training material was FrameNet Release 1.3 plus extra full-text annotations, including texts from the ANC. Testing used three new texts that FrameNet staff annotated by hand. Annotating them required many frames absent from training, so systems had to pick the nearest known frame. Scoring covered precision and recall in two ways: direct label matching, and comparison of dependency graphs built automatically from the annotations. Partial credit was given for frames and FEs linked to the gold answer through FrameNet's relation hierarchy. Three teams took part. Frame identification reached F1 up to about 0.75, while combined frame-plus-FE recognition was much harder, with F1 between about 0.35 and 0.55. The task matters because frame-based output supports richer inference than PropBank-style role labels.

Key points

  • Training data: FrameNet Release 1.3 has about 780 frames, about 10,000 lexical units and about 150,000 annotation sets. It was supplemented with ANC texts from Slate.com and a Berlitz guide, "History of Jerusalem".
  • Test set: three texts, "China Overview", "Work Advances" (both from NTI) and "Introduction to Dublin" (ANC/Berlitz). Together they contain 120 sentences and 1,059 frame tokens covering 272 frame types. Nearly 40 new frames were created while annotating them.
  • Participants: UTD-SRL (SVM and ME classifiers), LTH (SVM only) and CLR (hand-written rules, frames only). LTH kept the test data beyond the 10-day limit, so its results are not strictly comparable.
  • Scoring: a Perl tool, fttosem, converted annotations into XML semantic dependency trees. Partial credit for a related frame dropped by 20% for each link separating it from the gold frame in the relation graph.
  • Results: the best frame-only F1 was 0.7488 (LTH, "Work Advances"). Combined frame and FE F1 ranged from 0.352 to 0.546, and recall stayed low throughout. Dependency-based scoring generally gave higher scores than exact span matching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

55 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 44 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baker2007semeval,
  title = {{SemEval-2007 Task 19: Frame Semantic Structure Extraction}},
  author = {Baker, Collin and Ellsworth, Michael and Erk, Katrin},
  booktitle = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)},
  year = {2007},
  url = {https://aclanthology.org/S07-1018/}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Charles J. Fillmore and Collin F. Baker. 2001. Frame semantics for text understanding. In Proceedings of WordNet and Other Lexical Resources Workshop, Pittsburgh, June. NAACL. find
  2. Charles J. Fillmore, Christopher R. Johnson, and Miriam R.L Petruck. 2003. Background to FrameNet. International Journal of Lexicography, 16.3:235–250. find
  3. Charles J. Fillmore, Collin F. Baker, and Hiroaki Sato. 2004a. FrameNet as a “Net”. In Proceedings of LREC, volume 4, pages 1091–1094, Lisbon. ELRA. find
  4. Charles J. Fillmore, Josef Ruppenhofer, and Collin F. Baker. 2004b. FrameNet and representing the link between semantic and syntactic relations. In Chu-ren Huang and Winfried Lenders, editors, Frontiers in Linguistics, volume I of Language and Linguisitcs Monograph Series B, pages 19–59. Inst. of Li… find
  5. Charles J. Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280:20–32. find
  6. Charles J. Fillmore. 1982. Frame semantics. In Linguistics in the Morning Calm, pages 111–137. Hanshin Publishing Co., Seoul, South Korea. find
  7. Anette Frank and Jiri Semecky. 2004. Corpus-based induction of an LFG syntax-semantics interface for frame semantic processing. In Proceedings of the 5th International Workshop on Linguistically Interpreted Corpora (LINC 2004), Geneva, Switzerland. find
  8. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Rada Mihalcea and Phil Edmonds, editors, Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 9–12, Barcelona, Spain, July. Association for Computational Linguisti… find
  9. Sameer S. Pradhan, Wayne H. Ward, Kadri Hacioglu, James H. Martin, and Dan Jurafsky. 2004. Shallow semantic parsing using support vector machines. In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 233–240, Boston, Massachusetts, USA, May 2 - May 7. Assoc… find
  10. Jan Scheffczyk, Collin F. Baker, and Srini Narayanan. 2006. Ontology-based reasoning about lexical resources. In Alessandro Oltramari, editor, Proceedings of ONTOLEX 2006, pages 1–8, Genoa. LREC. find
  11. Sabine Schulte im Walde. 2003. Experiments on the choice of features for learning verb classes. In Proceedings of the 10th Conference of the EACL (EACL-03). find
  12. Steve Sinha and Srini Narayanan. 2005. Model based answer selection. In Proceedings of the Workshop on Textual Inference, 18th National Conference on Artificial Intelligence, PA, Pittsburgh. AAAI. find
  13. Suzanne Stevenson and Eric Joanis. 2003. Semi-supervised verb class discovery using noisy features. In Proceedings of the 7th Conference on Natural Language Learning (CoNLL-03), pages 71–78. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baker-2007-semeval-2007-task-19-frame/