📄 Mining the Web for Discourse Markers

Summary

Empirical work on discourse markers, and NLP systems that learn from them, need many example sentences. Even the 100-million-word BNC has very few instances of rarer connectives. Hutchinson proposes harvesting such sentences from the web through a five-stage pipeline:

• query a search engine for each marker's surface form;
• extract and sentence-split the page text;
• remove duplicate sentences;
• parse each sentence with Charniak's parser and keep only true connective uses;
• store the survivors in a database indexed by marker.

The pipeline was run for 116 structural connectives, with 8,000 pages per marker. On rare markers the web yielded far more examples than the BNC; for example, assuming that gave 4,695 sentences against 395. On a hand-checked sample, precision was 84.4–91.4%, while the agreement between the two judges suggests a human ceiling of 93%. Co-occurrence statistics of connectives with adverbial markers correlated strongly with BNC statistics. This suggests the web data reflects typical discourse contexts, making the web a practical way around data sparseness for discourse research.

Key points

• The setup used AltaVista, CPAN's HTML::Parser and Charniak's (2000) parser. It covered 116 structural connectives (e.g. and, after, as long as, given that, to the extent that), analysing 8,000 pages per marker.

• A syntactic test decides whether a candidate is a connective: it counts only if, in the parse tree, an S node comes immediately after it. The test tolerates some parser errors, such as an odd PP → IN S expansion.

• On five test markers, a random sample of 100 per marker was checked by two judges. System precision was 84.4–91.4%. The judges disagreed on 7% of cases, with overall kappa 0.671, but kappa was low for as long as. Marcu (1998) reported 89.5% on a similar task.

• Web vs BNC sentence counts:

  • as long as: 6,244 vs 2,357;
  • every time: 5,124 vs 660;
  • assuming that: 4,695 vs 395.

  Extrapolation via AlltheWeb page counts suggests millions to billions of tokens are available on the web, and these figures are lower bounds.

• Pearson correlation of connective–adverbial bigram counts between web and BNC was 0.80–0.96 for after, and, before, but and or. These values are higher than Keller and Lapata's web/BNC correlations for content-word bigrams.

Interesting ideas

• I1. Search engines cap the number of results per query, so the author splits each query into complementary halves ("X AND 1" / "X AND NOT 1"). This gets past the cap while assuming the extra term is roughly independent of X.
• I2. Exact duplicate sentences are dropped before analysis, so mirrored, syndicated or quoted text counts as one utterance. This saves compute and stops a single viral sentence from skewing distributional statistics.
• I3. Whether a word is a discourse use is decided from parse structure (the marker must sit right before a clause node) rather than from surface patterns. Recall is deliberately sacrificed for precision because the data source is effectively unlimited.
• I4. Representativeness of a new corpus is checked by correlating co-occurrence counts with a trusted balanced corpus. Here the counts were connective plus adverbial marker bigrams, which serve as a cheap proxy for discourse context.
• I5. Inter-judge disagreement is used as an explicit upper bound for system accuracy, and per-marker significance tests reveal which connectives are inherently hard (e.g. assuming that, as long as).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when harvesting silver sentences for a rare construction from any capped search API, partitioning queries this way multiplies the yield without bias-heavy reformulation.
• I2 (learn): deduplicating web-mined silver data before witness-parser agreement prevents one repeated sentence from looking like strong consensus evidence.
• I3 (tree): a UD-level rule such as "mark/cc attached to a clausal head (advcl, conj of a verb)" can separate connective uses of and, after or given that from phrasal ones, giving clean examples per marker.
• I4 (learn): the same correlation check against a reference treebank or corpus could validate any harvested silver set before training, acting as a distribution-level sanity control.
• I5 (frames): evaluating discourse-relation detection per connective, against a human-agreement ceiling, would show where prag's relation frames fail because of the system and where the ambiguity is inherent.