📄 Play the Language: Play Coreference

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P09-2053.pdf

Summary

Manually annotating coreference is slow and costly. For the Prague Dependency Treebank 2.0 (PDT 2.0), it took about two years. The authors propose PlayCoref, a two-player online game following the "Games With a Purpose" (GWAP) approach, in which players produce coreference annotation as a side effect of play. It is the first of several language games (alongside PlayNE and PlayDoc) planned for the LGame portal, and it is designed for Czech and English with language-independent rules.

Players read a document sentence by sentence and link words that refer to the same entity. Only nouns and certain pronouns can be linked, and links are undirected. Chains are rebuilt from the marked pairs afterwards. To allow evaluation against expert annotation, the first sessions use PDT 2.0 and MUC-6 texts. The PDT 2.0 coreference, which is annotated on the tectogrammatical layer, must first be projected onto surface words.

The paper presents a game design rather than experimental results. Its significance lies in showing how annotation that requires no linguistic expertise could be scaled up through crowd play.

Key points

• Motivation: PDT 2.0 coreference annotation ran from summer 2002 to autumn 2004, with two annotators splitting 3,165 documents. The authors estimate that 128 players could cover the same set in about 9 days, compared with 528 days for a single pair.
• Linkable words: words from eight Czech pronoun sub-POS classes cannot be linked: D and E, L and O, Q and W, and Y and Z. Examples include demonstratives, negatives and indefinites, because these often refer to spans longer than one word.
• PDT 2.0 projection: pairs without surface counterparts are dropped, for example zero subjects in pro-drop Czech. Grammatical and textual coreference types are merged. Extrapolated counts are roughly 96k pairs including extended textual coreference and about 70k in the surface subset. The extrapolation is based on about 200k of PDT 2.0's 833k tokens.
• Scoring: a player's points are a weighted sum of inter-coder agreement with an automatic coreference resolver and with the opponent. Agreement is measured by F-measure or Krippendorff's α and computed at the chain level. The score is shown only after the session ends.
• Opponent feedback: during play, each player sees only how many words the opponent has linked and in how many sentences.