🧵 Discourse analysis
Discourse analysis is the study of how stretches of language beyond the single sentence form coherent wholes. It covers the relations that link clauses and sentences (cause, contrast, elaboration), the structures those relations build, and the computational methods that recover them from text.
What it is
Discourse analysis asks what makes a sequence of sentences a text rather than a random list. A reader of "Max fell. John pushed him" understands that the pushing came first and caused the fall, even though nothing in the words says so. Discourse analysis tries to describe such links and to explain how readers infer them. The term has a broad life in linguistics and the social sciences, where it also covers conversation and ideology in talk. In computational linguistics it has a narrower, operational meaning: identifying the units of a text, the coherence relations among them, and the overall shape these relations form.
Three families of frameworks dominate the field:
| Framework | Unit and anchor | Structure | Main corpus |
|---|---|---|---|
| Rhetorical Structure Theory (RST) | clause-sized elementary discourse units (EDUs) | one tree per document, with nucleus/satellite roles | RST Discourse Treebank (RST-DT) |
| Penn Discourse TreeBank (PDTB) | connectives and their two arguments | flat, local relations with no global tree | PDTB on the Wall Street Journal |
| Segmented Discourse Representation Theory (SDRT) | EDUs and complex units | graphs, often non-tree | STAC dialogue corpus |
The key idea
The core mechanism is the discourse relation. It is a typed link between two spans of text, each describing an abstract object such as an event, a state or a proposition. A connective may signal the relation overtly ("because", "however"), or the reader may have to infer it. Most automatic systems treat recognising a relation as choosing the most probable sense given its arguments:
ŝ = argmax over s in S of P(s | Arg1, Arg2, c)
Here S is the inventory of relation senses, for example the four PDTB top-level classes Comparison, Contingency, Expansion and Temporal. Arg1 and Arg2 are the two text spans the relation joins. Arg2 is the span syntactically tied to the connective. The variable c is the connective, which may be empty when the relation is implicit, and ŝ is the predicted sense. In RST-style analysis, the same decision is applied recursively. Adjacent spans are merged into larger spans, and each merge receives a relation label and a nuclearity assignment, which marks which span is more central. The result is a tree covering the whole document.
The presence or absence of c is the field's central practical divide. Explicit connectives are only mildly ambiguous. A classifier that looks only at the connective labels explicit PDTB relations with 93.09% accuracy on the four top-level senses (Pitler et al. 2008). Without a connective, the system must rely on lexical meaning, syntax, entities and context, and accuracy drops sharply. On the finer set of 11 second-level PDTB types, an early implicit classifier reached 40.2% accuracy, against a majority baseline of about 26.1% (Lin et al. 2009). The two label sets differ, so these figures are not directly comparable.
History and founding papers
The phrase "discourse analysis" goes back to Zellig Harris (1952). Theoretical foundations for coherence relations came from Hobbs (1985) and from Grosz and Sidner (1986) on intentional structure. RST was developed by Mann and Thompson during the 1980s, and their 1988 paper is its standard statement. Early formal work showed that discourse relations could be inferred through default reasoning that combines world knowledge with pragmatic maxims. One example is narrating events in the order they happened unless causal knowledge overrides it (Lascarides & Asher 1991). RST quickly drew criticism. Moore and Pollack argued that text segments relate simultaneously on an informational level and an intentional level, and that a single relation per pair cannot capture both (Moore & Pollack 1992).
The empirical turn came with large annotated corpora. The RST Discourse Treebank (RST-DT, released as the RST Corpus) layered rhetorical trees over 385 Penn Treebank WSJ articles: more than 176,000 words segmented into 21,789 EDUs and labelled with 78 relations. Annotator agreement was highest for segmentation and declined higher up the tree (Carlson et al. 2001). Taking a different approach, the PDTB anchored relations in lexical connectives and their arguments. It deliberately avoided committing to any one theory of global structure (Miltsakaki et al. 2004). Its 2.0 release covers about one million words and includes 18,459 explicit connective tokens. Annotators also inserted implicit connectives between adjacent sentences and recorded attribution (Prasad et al. 2008). Webber and colleagues supplied a theoretical rationale for this lexical approach. They argued that discourse adverbials behave anaphorically, like pronouns, while conjunctions link structurally adjacent units (Webber et al. 2003). Wolf and Gibson annotated 135 news texts with high agreement (Kappa, a chance-corrected agreement measure, above 0.8). They found frequent crossed dependencies and multi-parent nodes, which challenges the assumption that discourse is tree-shaped (Wolf & Gibson 2005).
How it is used today
Most current work is computational. The bulk of it follows the two large corpus traditions, RST and PDTB. SDRT work has centred on dialogue, where its graph structures matter most (Asher et al. 2016).
- RST parsing. Soricut and Marcu showed that sentence-internal trees can largely be derived from syntax. Given gold syntax and gold segmentation, their parser came within about 1.5 F points of human agreement. Perfect segmentation helped more than perfect syntactic parses, which marks segmentation rather than syntactic parsing as the main bottleneck (Soricut & Marcu 2003). Document-level parsing proved harder. Rich features help, but performance drops sharply for relations spanning sentences (Feng & Hirst 2012). Joty et al. combined intra- and multi-sentential CRFs with optimal bottom-up search (Joty et al. 2013). Representation learning followed. A shift-reduce parser over a learned latent space raised labelled relation F-measure on RST-DT by roughly 6 points over the prior best system, that of Joty et al. (2013) (Ji & Eisenstein 2014). Recursive neural models scored 84.0 span, 70.8 nuclearity and 58.6 relation F, on par with contemporary parsers without leading on any single metric (Li et al. 2014). A two-stage design that first builds an unlabelled tree and then labels relations scored 86.0 span, 72.4 nuclearity and 59.7 relation F on RST-DT (Wang et al. 2017). For comparison, reported human agreement is 88.70, 77.72 and 65.75 (Joty et al. 2013).
- Implicit relation recognition. Marcu and Echihabi generated training pairs by stripping cue phrases from about a billion words of text (Marcu & Echihabi 2002). Later work showed that such synthetic examples transfer poorly to naturally implicit relations, and it evaluated on the PDTB's natural distribution instead (Pitler et al. 2009). Successive gains came from:
- predicting the missing connective with a language model (Zhou et al. 2010);
- feature selection (Park & Cardie 2012);
- Brown cluster pairs and coreference patterns (Rutherford & Xue 2014);
- entity-augmented neural representations (Ji & Eisenstein 2015);
- multi-task learning across PDTB, RST-DT and noisy news data (Liu et al. 2016).
- Shared benchmarks. The CoNLL-2015 shared task defined end-to-end shallow discourse parsing. Sixteen teams took part. The best system reached only 24.0% end-to-end parser F1, which scores connectives, arguments and senses together, on a blind Wikinews test set (Xue et al. 2015). The 2016 edition added Chinese and drew 23 teams (Xue et al. 2016).
- Downstream applications. Discourse structure has served as a signal for several tasks:
- evaluating coherence (Lin et al. 2011);
- review summarization (Gerani et al. 2014);
- sentiment analysis, where a recursive network over RST trees reached 84.1% accuracy on the Pang and Lee (2004) set of 2,000 reviews and 85.6% on a 50,000-review set, against 82.4% and 81.5% for a bag-of-words classifier (Bhatia et al. 2015);
- text categorization (Ji & Smith 2017);
- argumentation mining (Peldszus & Stede 2015).
- Dialogue. The STAC corpus annotates multi-party game chats with full SDRT graphs and links verbal moves to non-verbal game events (Asher et al. 2016).
Open questions
Several issues remain unsettled.
Structure is still contested: tree, flat relations or general graph. Corpus evidence of crossing dependencies (Wolf & Gibson 2005) and dialogue structures that trees cannot represent (Asher et al. 2016) argue for graphs. Anaphoric treatment of adverbials offers a way to keep a simpler backbone (Webber et al. 2003).
Moore and Pollack's informational/intentional split implies that one label per pair of segments is not enough (Moore & Pollack 1992). The RST-style scheme described above still gives each tree node a single rhetorical relation (Carlson et al. 2001).
Implicit relations remain the weakest component. Error analyses point to ambiguity between senses, the need for inference, wider context and world knowledge (Lin et al. 2009). Parser errors also limit the downstream uses that depend on implicit relations (Lin et al. 2011).
Data remain scarce, and for English the two largest corpora are both built on WSJ news. News-trained discourse parsers transfer poorly to other genres. Legislative bills are one example: there, a discourse-based text classifier failed to beat earlier systems, which its authors attributed to the parser's training domain (Ji & Smith 2017). Errors also compound across pipeline stages (Xue et al. 2015).
Finally, cross-sentence relations are where document-level parsing falls furthest short (Feng & Hirst 2012). As of 2017, even the best parsers remained below the human agreement ceiling on relation labels (Wang et al. 2017).
Key papers
Related concepts
- Coreference resolution — Coreference resolution is the task of finding the expressions in a text that refer to the same entity as one another, such as names…
- Implicature — Implicature is the part of what a speaker communicates that goes beyond the literal content of the words. Hearers infer it by assuming the…
- Semantic role labeling — Semantic role labeling is the task of finding, for each predicate in a sentence, the phrases that act as its participants and labeling each…
- Corpus linguistics — Corpus linguistics is the empirical study of language through large, principled collections of naturally occurring text or transcribed…
More papers
Topics: 🧵 Discourse analysis · 🔗 Coreference resolution
Shelves: 🗣️ Linguistics & languages
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/discourse-analysis/