The Penn Discourse Treebank
What it is about
The paper introduces the PDTB project. It adds a discourse layer to the Penn Treebank and Propbank: discourse connectives, their two arguments (Arg1 and Arg2), and, for implicit relations, an inserted connective. Connectives are of four kinds: subordinating and coordinating conjunctions, adverbials, and implicit ones. The paper reports a pilot. Two annotators marked 2,717 tokens of 10 explicit connectives and 386 implicit tokens. For the argument spans of explicit connectives, exact-match agreement is 90.2% (4,900 of 5,434 argument tokens). It is 92.4% for subordinating conjunctions and 71.8% for adverbials. Of the disagreements, 79% are partial overlaps, mostly over a governing verb, a dependent clause or a parenthetical. For implicit connectives, argument agreement is 85.1% (657 of 772 tokens), and agreement on the class of the inserted connective is 72% (307 tokens). The authors do not use kappa, because the spans do not fall into fixed categories.
Ideas
- I1. Report two agreement numbers for any Mova annotation: exact span and partial overlap. The paper's partial-match figure (94.5%) shows how much the choice of measure changes the result.
- I2. Store each argument as a minimal span and a separate supplementary span. A rule can then use the minimal span for sense and the full span for retrieval. The same split fits the frames in
prag/src/frames.rs. - I3. Use five coarse classes for inserted connectives (additive, cause-effect, temporal, contrast, restatement), as the paper does, and measure agreement on the same kind of sample.
- I4. Report the adverbial and subordinating classes apart. The gap in the paper (92.4% vs 71.8%) shows where a rule-based argument finder is likely to fail, and the same split can be measured on Mova's output.
For Mova
The chapters are languages/en/36-discourse-relations.md and languages/en/38-dialogue-and-discourse-markers.md. The code is prag/src/frames.rs. The measure is a two-annotator agreement check on a sample of Mova's own lesson annotations, with the two span measures and the per-class split above. Run it twice. The paper is a 2004 pilot with small counts, so the numbers are a guide, not a target.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/L04-1386.pdf
Summary
Discourse-level annotated resources at the time either focused on other phenomena, such as coreference or temporal relations, or, like the RST corpus, assigned rhetorical relations without an explicit textual basis. The authors introduce the Penn Discourse Treebank (PDTB), a corpus project that anchors discourse structure in lexical connectives and the two text spans (arguments) each connective links. The annotation is layered over the Penn Treebank and Penn Propbank, so syntactic, semantic and discourse information can be combined when building and testing NLP systems. Four connective classes are covered: subordinating conjunctions, coordinating conjunctions, adverbials, and implicit connectives between adjacent sentences. For implicit relations, annotators also supply the explicit connective that would best express the inferred relation. Arguments must denote abstract objects, such as events or states. A preliminary agreement study on 10 explicit connectives and a set of implicit ones, using a strict exact-match criterion, yields high agreement. This suggests that connective-based discourse structure can be annotated reliably and is worth scaling up.
Key points
- The planned corpus contains about 30,000 annotations: 20,000 covering the 250 explicit connective types found in the corpus and 10,000 implicit ones. Release was expected by November 2005, and annotation was done in the WordFreak tool.
- The tags are CONN, ARG1 and ARG2, plus optional SUP1/SUP2 tags for supplementary spans that help interpret an argument.
- On explicit connectives (2,717 tokens, 5,434 argument spans), exact-match agreement reached 90.2%.
- Subordinating conjunctions scored 92.4%.
- Anaphoric adverbials scored 71.8%.
- A partial-match measure raises the overall figure to 94.5%.
- Partial overlap accounted for 79% of the disagreements. Typical cases were inclusion of a higher verb, a dependent clause, or a parenthetical.
- Results for implicit connectives (386 tokens):
- Argument agreement was 85.1% (657/772), rising to 92.6% with partial match.
- Agreement on the type of the supplied connective was 72%, measured across five relation groups: addition, cause, temporal, contrast, and restatement.
Cite
@inproceedings{miltsakaki2004penn,
title = {{The Penn Discourse Treebank}},
author = {Miltsakaki, Eleni and Prasad, Rashmi and Joshi, Aravind and Webber, Bonnie},
booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
year = {2004},
doi = {10.63317/2rrr3zi7vbno},
url = {https://doi.org/10.63317/2rrr3zi7vbno}
}References
- Asher, Nicholas, 1993. Reference to Abstract Objects in Discourse. Kluwer Academic Publishers. find
- Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski, 2003. Current Directions in Discourse and Dialogue, chapter Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory. Kluwer Academic Publishers. find our card
- Gaizauskas, Rob, Patrick Hanks, James Pustejovsky, Roser Sauri, Andrew See, Andrea Setzer, Lisa Ferro, and Beth Sundheim., 2003. The timebank corpus. In Corpus Linguistics 2003.. Lancaster, U.K. find
- Kingsbury, Paul and Martha Palmer, 2002. From Treebank to Propbank. In Third International Conference on Language Resources and Evaluation, LREC-02, Las Palmas, Canary Islands, Spain. find our card
- Knott, Alistair, 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh. find
- Marcus, Mitch, Beatrice Santorini, and Mary Ann Marcinkiewicz, 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19:313–330. find our card
- Miltsakaki, Eleni, Rashmi Prasad, Aravind Joshi, and Bonnie Webber, 2004. Annotating discourse connectives and their arguments. In Proceedings of the Workshop on Frontiers in Corpus Annotation. Boston, Massachussetts. find
- Mitkov, Ruslan, R. Evans, C. Orasan, C. Barbu, L. Jones, and V. Sotirova, 2000. Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies. In Proceedings of the Discourse Anaphora and Anaphora Resolution Colloquium (DAARC 2000), Lancaster, U.K.. find
- Muller, Christoph, Stefan Rapp, and Michael Strube, 2002. Applying co-training to reference resolution. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Philadelpha PA. find
- Siegel, Sidney and N. J. Castellan, 1988. Nonparamateric Statistics for the Behavioral Sciences. McGraw-Hill, 2nd edition. find
- Webber, Bonnie, Matthew Stone, Aravind Joshi, and Alistair Knott, 2003. Anaphora and discourse structure. Computational Linguistics, 29:545–587. find our card