📄 SemEval-2010 Task 10: Linking Events and Their Participants in Discourse

Summary

Semantic role labeling usually stops at the sentence boundary. As a result, it ignores participants that a predicate leaves unexpressed but that the reader can recover from nearby text. This SemEval-2010 shared task targets that gap. Systems must find "null instantiated" core roles, decide whether each omission is definite (DNI) or indefinite (INI), and link DNIs to antecedents elsewhere in the discourse. The organizers annotated two Arthur Conan Doyle stories in FrameNet style, with null instantiations and coreference, and also provided a semi-automatic PropBank mapping. The public-domain status of these texts let the data be released freely. Because frame-semantic parsing of this new domain proved hard, a plain SRL track was added alongside the full and NI-only tasks. Results were modest. The best SRL system, SEMAFOR, reached an argument-recognition F1 of about 0.54, and telling DNI from INI barely beat a majority baseline. The task matters because it was an early attempt to bring discourse context into SRL, which could help information extraction, QA and summarization.

Key points

• Data sources: Training data comes from "The Tiger of San Pedro", the second part of "The Adventure of Wisteria Lodge". Test data is drawn from the two closing chapters of "The Hound of the Baskervilles". Formats were TIGER/SALSA XML (FrameNet-style) and a modified CoNLL format (PropBank-style); every participant chose FrameNet-style.
• Data size: The training set has 438 sentences, 1,370 frame instances and 303 DNIs (245 resolved). The test set has 525 sentences, 1,703 frame instances, 349 DNIs (259 resolved) and 361 INIs.
• How common omissions are: Roughly 18% of frame elements are omitted. Most DNIs have an antecedent in the text (80.9% in training, 74.2% in test). Pairwise annotator agreement on frame assignment was 0.74–0.79 F.
• SRL track (argument recognition F1): SEMAFOR scored 0.5448, the Shalmaneser reference parser (trained on FrameNet 1.3) scored 0.4812, and CLR scored 0.1921. The organizers supplied FrameNet 1.4 alpha as an intermediate release.
• NI track: There were 710 gold null instantiations in total. SEMAFOR detected 63.4% of them and GETARUNS++ detected 8.0%. Correct DNI/INI typing among detected cases was 54.7% for SEMAFOR and 64.2% for GETARUNS++, against a majority baseline of 50.8%.

Interesting ideas

• I1. Head-based scoring plus an overlap measure: A predicted antecedent counts as correct if its span contains the head of the gold filler. A second, overlap-based score stops systems from gaining by predicting huge spans. This pairing suits any span-linking evaluation where exact boundaries are fuzzy.
• I2. Choosing the genre on purpose: Linear-narrative fiction was picked because it should contain more omissions resolvable from context than news, which has an "inverted pyramid" structure. Using copyright-free texts also made redistribution easy. Matching the genre to the phenomenon, and the source to licensing needs, is a useful rule for building new corpora.
• I3. Automatic consistency checking: A software tool checked annotations against FrameNet's frame and frame-element specifications and warned annotators about violations. This complemented double annotation, adjudication and a shared wiki, and could be reused in any schema-heavy annotation project, such as UD treebanks.
• I4. Scoring pipeline stages separately: The organizers concluded that the task should have reported separate scores for (1) detecting a null instantiation, (2) classifying it as DNI or INI, and (3) finding the antecedent. Per-stage scoring gives clearer diagnostics for new multi-step tasks than a single end-to-end number.
• I5. Modeling dependencies between roles: FrameNet's CoreSet relations mean omissions cannot be judged one role at a time. Systems that ignore these relations produce false DNIs. Constraints between labels should therefore be modeled jointly, not as independent decisions.