Anaphoric Annotation in the ARRAU Corpus

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/L08-1091.pdf

Summary

Most anaphora resources cover only newspaper prose, mark a narrow set of relations, and assume every anaphor has one clear nominal antecedent. Poesio and Artstein present ARRAU, an English corpus built at the University of Essex (2004–2007) that tackles two neglected phenomena: anaphors whose antecedent is genuinely ambiguous, and discourse deixis, where an expression points back to an event, action or plan introduced by a clause rather than a noun phrase.

Multi-annotator pilot studies guided the design of the annotation scheme. The texts span task-oriented dialogue (Trains-91/93), spoken narrative (Pear Stories), the Gnome corpus and Wall Street Journal articles from the Penn Treebank. Annotation was done in MMAX2 and covers:

agreement features, grammatical function and animacy/category for every NP;
antecedent pointers, with two alternative pointer sets for ambiguous items;
links from NPs to clauses for abstract reference;
selected bridging relations.

Because ambiguity is allowed, the annotations are not simple equivalence chains. The authors argue that ARRAU is the richest linguistically oriented anaphora resource available. It is also usable for evaluating anaphora resolvers and was first used at the 2007 JHU ELERFED workshop.

Key points

Pilot studies: up to 20 coders annotated independently. Krippendorff's α reached about 0.6–0.7 on anaphoric chains and about 0.55 on discourse deixis, where coders agreed on the region but not its exact boundaries.
Size: Table 1 lists 110 texts. Summed across sources, this gives roughly 94K words and about 24K markables.
- About 10K markables have a nominal antecedent.
- 455 point to text regions.
- 103 are explicitly flagged as ambiguous.
Largest component: the WSJ portion has 50 texts, 32,771 words and 9,177 markables. Most of these texts also belong to the RST Discourse Treebank, which is being annotated jointly with Kibrik's group in Moscow.
Preprocessing and bridging: NPs and clauses not already marked were found with the Charniak parser and then hand-corrected. Bridging annotation covers part-of, set membership and a converse relation.
Workshop version: for ELERFED, ambiguous anaphors were reduced to their first-marked reading to produce equivalence sets. The WSJ data was extended with 35 Vieira-Poesio texts and 40 Moscow-annotated texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Coreference resolution

Cited by

76 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{poesio2008anaphoric,
  title = {{Anaphoric Annotation in the ARRAU Corpus}},
  author = {Poesio, Massimo and Artstein, Ron},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/2s5fn4tnkxk5},
  url = {https://doi.org/10.63317/2s5fn4tnkxk5}
}

References

2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

This work builds on several years of experience with anaphoric annotation summarized in Poesio (2004b). The annotation experiments discussed in Poesio and Vieira (1998) were primarily concerned with establishing whether annotation could reliably capture a distinction between coreference, bridging, and discourse novelty inspired by the proposals of Prince (1992), using newspaper articles from the Wall Street Journal portion of the Penn Treebank as data. These studies revealed the difficulty of annotating reliably the whole range of bridging relations, and distinguishing those cases from discourse-new. The result was the so-called Vieira-Poesio corpus of 34 documents, which was used in a number of subsequent studies. Subsequent work on the GNOME corpus (Poesio, 2004a) focused on identifying a subset of bridging relations that could be reliably annotated, as well as annotating other types of information that could be useful in the study of anaphora in general and local salience in particular, such as agreement features, grammatical function, and animacy. The GNOME coding instructions allowed for ambiguity, but this feature was not systematically studied. Limited progress was made towards annotating discourse deixis, marking only the type of antecedent of abstract anaphora in an attempt to distinguish between references to events, temporal objects, propositions, and types (Poesio and Modjeska, 2005). In subsequent work on the VENEX corpus (Poesio et al., 2004) we carried out preliminary studies of annotation of coreference in dialogue, including annotation of references to objects in the visual situation, and started experimenting with an early version of the MMAX tool (Müller and 2 http://www.clsp.jhu.edu/ws2007/groups/elerfed/ ↗
Strube, 2006). 3. Agreement on ambiguity and discourse deixis In order to determine the best way to annotate anaphoric ambiguity and discourse deixis in Arrau we conducted a series of experiments, some with pen and paper, but mostly using an annotation tool, MMAX2 (Müller and Strube, 2006), which forced the coders to use a predefined scheme. In these annotation experiments, multiple annotators (as many as 20) worked independently on the same text, and formal reliability measures such as α (Krippendorff, 1980) were used to compare the annotations and identify easy and difficult parts of the task; agreement on anaphoric chains was in the range of α ≈ 0.6–0.7 (Poesio and Artstein, 2005b). We found that while annotators often missed the ambiguity of an item, it was possible to identify ambiguity implicitly when sets of annotators chose different antecedents for a single item (Poesio and Artstein, 2005a). For discourse deixis we found that annotators agreed on the general textual regions that evoke the referents, though they often disagreed on the exact boundaries, resulting in agreement of around α ≈ 0.55 (Artstein and Poesio, 2006). The experiments led to progressive refinements of the annotation scheme to one which was more reliable yet expressed the distinctions of interest. We clarified the distinction between multiple antecedents for plural expressions and for ambiguous expressions, we added a way to mark ambiguity between discourse-old and discourse-new interpretations, and we constrained the marking of textual regions to predefined clause-level units. The resulting scheme is described in section 4.2. 4. The corpus 4.1. Composition The ARRAU corpus contains texts from a mixture of genres, including dialogue, narrative, and a variety of genres of written text. Task-oriented dialogues include texts from the Trains-91 and Trains-93 corpora (Gross et al., 1993; Heeman and Allen, 1995). Spoken narratives include the full English Pear Stories corpus of Narratives (Chafe, 1980). 3 Examples of written text include five texts from the Gnome corpus not yet annotated for anaphoric relations and – the final and largest part of the corpus – newspaper text from the Wall Street Journal portion of the Penn Treebank (Marcus et al., 1993). Most of the WSJ texts are part of the RST Discourse Treebank (Carlson et al., 2003) and are being annotated as part of a joint effort with prof. Kibrik’s group at the the Russian Academy of Sciences in Moscow. The current composition of the corpus is summarized in Table 1. It is expected that the final version of the corpus will also include half of the texts in the RST discourse treebank, as well as the files from the Vieira-Poesio corpus and the previously annotated files in the GNOME corpus. 4.2. Annotation scheme The corpus was created using the MMAX2 tool (Müller and Strube, 2006), which allows marking text units at different 1171 find