The CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue
What it is about
The shared task asks systems to find anaphoric relations in conversation, a genre that OntoNotes-style corpora barely cover. Three tasks are set: identity coreference (including split-antecedent plurals), bridging references, and discourse deixis. The data are four newly annotated dialogue corpora (AMI meetings, LIGHT fantasy-game dialogues, PERSUASION conversations, SWITCHBOARD telephone talks) with the ARRAU scheme, plus the TRAINS dialogues from ARRAU for training. The new part has 147,725 tokens and 41,807 markables in total. Annotation is in MMAX with mentions from a mention detector, and scoring uses the Universal Anaphora CoNLL-U extension and its scorer. The paper reports 115 runs from six teams, with most systems beating the baselines. This card rests on the paper's introduction, data and task sections; I did not verify the per-task results tables.
Ideas
- I1. Dialogue markables for Mova: count the discourse-deictic cases ("that", "this" referring to an event or fact) in dialogue lessons in
coref/, and score them with a separate measure, since the identity scorer cannot see them. - I2. Split-antecedent plurals as a rule in
coref/src/sieve.rs: when a plural pronoun ("they", "both") has no single plural antecedent, search for two singular antecedents in the same sentence or the previous one and link them as one plural entity, reporting how many such links fire. - I3. Keep the scorer separate:
coref/src/eval.rsshould report identity, bridging and discourse deixis as three numbers on a dialogue set, so that a gain on identity is not read as progress on the other relations.
For Mova
Code: coref/src/sieve.rs (split antecedents, discourse deixis), coref/src/eval.rs (per-relation scores), coref/src/corefud.rs (CoNLL-U with anaphoric extensions). Chapters: languages/en/38-dialogue-and-discourse-markers.md, languages/en/36-discourse-relations.md, languages/en/35-reference-and-coreference.md. Measure: per-relation F1 on one dialogue set (not used for tuning), run twice, and a negative control with turns shuffled, which should lower the score.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.codi-sharedtask.1.pdf
Summary
Coreference and anaphora resolution has advanced mainly on written genres such as news (ONTONOTES). Conversation brings disfluencies, frequent deixis, vague abstract reference and speaker-dependent pronouns, and it has remained largely unexplored for English, mostly because suitable annotated corpora were missing. The CODI-CRAC 2021 shared task targets this gap.
The organizers used five dialogue datasets. Four of them (LIGHT, PERSUASION, SWITCHBOARD, AMI) were newly annotated for identity anaphora, bridging and discourse deixis and served as development and test data. ARRAU sub-corpora were provided for training, so the setup made transfer learning and out-of-domain generalization central. Three separate tasks were defined, and bridging and discourse deixis were scored in both gold-mention and predicted-mention settings. Evaluation used the Universal Anaphora scorer.
Six teams submitted 115 runs, and most clearly outperformed the baselines. The strongest identity-resolution system gained roughly 30 CoNLL Avg. F1 points over the baseline. The results suggest identity coreference in dialogue is becoming tractable, while bridging and discourse deixis remain difficult. This motivated a second edition with new test data.
Key points
- Tasks: three tasks covered identity anaphora, bridging and discourse deixis, with separate leaderboards for gold and predicted mentions. Results were reported per sub-corpus (LIGHT, PERSUASION, SWITCHBOARD, AMI).
- Task 1 (identity): five teams submitted 36 runs.
- The Emory team (lxucs) ranked first on every sub-corpus. Its CoNLL Avg. F1 scores were around 75% or higher for LIGHT, PERSUASION and SWITCHBOARD.
- AMI was hardest, at roughly 64%, because its conversations are long and ambiguous.
- Task 2 (bridging): three teams took part, evaluated by Entity F1.
- UTD_NLP led the predicted setting, scoring 21.92 on PERSUASION but only about 10–15 on the other sub-corpora.
- In the gold setting, UTD_NLP scored 31.40 and INRIA 16.28 on PERSUASION.
- Task 3 (discourse deixis): 25 runs were submitted.
- In the predicted setting, UTD_NLP reached about 35–42 CoNLL Avg. F1, almost double the second team (DFKI) and far above the baseline (about 12–18).
- Infrastructure: the task reused the CRAC 2018 ARRAU shared-task design and used a new Universal Anaphora scorer. The Universal Anaphora markup extends the CONLL-U format from Universal Dependencies.
Cite
@inproceedings{khosla2021codi,
title = {{The CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue}},
author = {Khosla, Sopan and Yu, Juntao and Manuvinakurike, Ramesh and Ng, Vincent and Poesio, Massimo and Strube, Michael and Rosé, Carolyn},
booktitle = {Proceedings of the CODI-CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue},
year = {2021},
doi = {10.18653/v1/2021.codi-sharedtask.1},
url = {https://doi.org/10.18653/v1/2021.codi-sharedtask.1}
}