📄 The CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis Resolution in Dialogue: A Cross-Team Analysis

Summary

CODI-CRAC 2021 was the first shared task devoted entirely to anaphora resolution in dialogue. It had three tracks: entity coreference, bridging, and discourse deixis. The last two are rarely studied, especially for conversational data. This paper compares the participating systems across teams, combining statistics computed from their outputs with manual inspection of their errors. For entity coreference, the authors break mention extraction down by pronoun type and compare official scores on four dialogue corpora. For discourse deixis, they argue that structural limits on antecedents shrink the search space, so the real bottleneck is deciding whether a word like "that" or "this" is deictic at all. The authors stress that deep error analysis is hard because teams submitted clusters rather than links, models are tightly integrated, and teams were free to use external data. They recommend that future shared tasks collect system links, run diagnostic tests with gold mentions or anaphors, and check annotation quality.

Key points

• Four teams were analysed in the entity coreference track: Emory, UTD, KU and DFKI. They were evaluated on the LIGHT, AMI, Persuasion and Switchboard corpora using MUC, B³, CEAFe and CoNLL scores. Emory scored best overall, and AMI was the hardest corpus.
• Every system except DFKI averaged above 85% mention-extraction F-score. Emory and KU favoured recall, while UTD had the highest precision.
• About half of the 19,051 gold mentions are pronouns, reflecting dialogue style. Roughly 80% of wrongly extracted mentions are non-pronominal phrases, and only about 4% of gold mentions were missed by all systems.
• Discourse deixis antecedents are limited to utterances within one turn and tend to appear close to the anaphor. Recognising the anaphor is the main difficulty, and DFKI over-labelled "this" and "that" as deictic.
• The Gold phase supplied a confusing union of mentions from all three tracks. In the deixis track, these gold mentions also omitted the antecedents.