Elamur Research⌕
☰
home / papers / 2020 / Model-based Annotation of Coreference

Model-based Annotation of Coreference

Summary

Standard coreference annotation asks people to connect pronouns with earlier text spans. The authors argue this is unnatural, because readers reason over a mental model of the entities a text describes rather than over strings. They propose model-based annotation instead. Documents are first linked to a knowledge base, and annotators attach each pronoun to one or more entries from a pre-extracted entity list. They can also add missing entities or mark a pronoun as No Reference.

To test the idea, they collected 200 English documents: 100 Wikipedia summary paragraphs and 100 teacher-student dialogues from QuAC. They then ran two Amazon Mechanical Turk setups, a grounded task showing the entity list and a conventional free span-selection task. The grounded setup produced clearly higher inter-annotator agreement and, according to the authors, faster annotation. The data also forms two new benchmarks on which existing resolvers perform weakly.

The approach matters because it reduces annotator effort and provides cluster representatives from the outset. It may also extend to ellipsis, implicit arguments or frame roles.

Key points

  • Corpus composition: the 100 Wikipedia documents are summaries containing at least five pronouns, of which 30 come from WikiCoref titles and 70 were picked at random. The 100 QuAC items are the first datapoints of that dataset. Entities were harvested from Wikipedia and Wikidata links.

  • Annotation design: only pronouns were markable. In each source, 30 documents were doubly annotated and 70 singly. Workers had to be native English speakers with an approval rate above 90%. Every fifth document was a hidden test item, and workers needed over 90% agreement with gold to continue.

  • Inter-annotator agreement (exact match / F1):

    SourceGroundedFree span
    Wiki0.70 / 0.740.50 / 0.65
    QuAC0.65 / 0.670.52 / 0.64

  • Baseline results (macro-averaged MUC, B³ and CEAFφ4 F1):

    SystemWikiQuAC
    Clark and Manning (2016)27.9523.39
    Lee et al. (2018)26.6723.79
    Aralikatte et al. (2019), using gold mention brackets45.9646.14

  • Availability: the data and the baseline coreference links are released at github.com/rahular/model-based-coref.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1906.10724

@inproceedings{aralikatte2020model,
  title = {{Model-based Annotation of Coreference}},
  author = {Aralikatte, Rahul and Søgaard, Anders},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  eprint = {1906.10724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.10724}
}

References

8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aralikatte, R., Lamm, M., Hardt, D., and Søgaard, A. (2019). Ellipsis and coreference resolution as question answering. CoRR, abs/1908.11141. — [arXiv:1908.11141](https://arxiv.org/abs/1908.11141) find
  2. Blackburn, P. and Bos, J. (2005). Representation and Inference for Natural Language. find
  3. CSLI, Stanford, CA. Bower, G. and Morrow, D. (1990). Mental models in narrative comprehension. Science, 247(4938):44–48. find
  4. Cai, J. and Strube, M. (2010). Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of the SIGDIAL 2010 Conference, pages 28–36. Association for Computational Linguistics, Sep. find our card
  5. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-t., Choi, Y., Liang, P., and Zettlemoyer, L. (2018). QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium, October-November. Association for… find our card
  6. Clark, K. and Manning, C. D. (2016). Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256– 2262, Austin, Texas, November. Association for Computational Linguistics. nouns in picture n… find our card
  7. Webster, K., Recasens, M., Axelrod, V., and Baldridge, J. (2018). Mind the gap: A balanced corpus of gendered ambiguous pronouns. In Transactions of the ACL, page to appear. find our card
  8. Yimam, S. M., Gurevych, I., de Castilho, R. E., and Biemann, C. (2013). Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of ACL. find

This page in Markdown → https://md.research.elamur.ai/free/papers/aralikatte-2020-model-based-annotation-coreference/