📄 Contrasting the Automatic Identification of Two Discourse Markers in Multiparty Dialogues

Summary

The paper addresses a binary decision task: is a given occurrence of like or well a discourse marker (DM) or an ordinary word? Both items are highly ambiguous and appear often in conversation. The authors annotated every occurrence of the two words in the ICSI Meeting Recorder Corpus, which contains 75 multi-party meetings, about 80 hours of audio and roughly 800,000 transcribed words. That yields more than 4,000 instances per item, far more than earlier studies, which often spread about 1,000 tokens over 30–40 DM types.

Using only surface cues, they trained C4.5 decision trees and other learners. The cues were:

• neighbouring words,
• utterance position,
• simple timing measures,
• speaker metadata.

The best accuracy is around 90%, with κ near 0.78. This beats earlier reported results and matches the agreement measured among human non-expert annotators.

Two findings stand out:

• Separate classifiers for each word work clearly better than a single pooled model.
• Lexical collocations, especially the word just before the candidate, carry most of the signal.

The work matters because DM detection is a prerequisite for tagging, parsing and understanding spontaneous dialogue.

Key points

• Data:
  • ICSI Meeting Recorder Corpus, 75 meetings with 5–8 speakers, including native and non-native speakers.
  • like has 4,519 occurrences, of which 2,052 (45%) are DMs.
  • well has 4,136 occurrences, of which 3,639 (88%) are DMs.
• Features:
  • Lexical window features: WORD(−N)…WORD(+N), with a frequency cutoff F.
  • Positional features: INITIAL, FINAL.
  • Temporal features: PAUSE-BEFORE, PAUSE-AFTER, DURATION, taken from forced alignment.
  • Sociolinguistic features: GENDER, AGE, EDUCATION, NATIVE, ORIGIN.
  • TYPE, which marks whether the item is like or well.
• Best combined result (Bayesian Network):
  • CCI = 90%, κ = 0.78.
  • Recall 0.96, precision 0.90, f = 0.93.
  • C4.5 and SVM scores were only slightly lower.
• Feature selection:
  • CFS selected {TYPE, PAUSE-BEFORE, INITIAL, WORD(−1)}.
  • Lexical features alone come within 5% of the best overall scores.
  • Sociolinguistic features help only marginally, and mostly for like.
• Human ceiling: Non-expert inter-annotator agreement was κ = 0.74. An earlier trivial baseline in Siegel and McKeown's study, using only an utterance-initial feature, already reached 79.16% accuracy.

Interesting ideas

• I1. Train one classifier per ambiguous lexical item instead of one model for the whole DM class, since the useful cues differ from word to word.
• I2. Treat human inter-annotator agreement as the upper bound on achievable scores, and report a chance-corrected κ rather than raw accuracy when classes are skewed (88% of well tokens are DMs).
• I3. Compare against very simple baselines, such as a single positional feature, because they can already score high and make sophisticated models look better than they are.
• I4. Run attribute selection on individual neighbouring words to surface readable collocation cues, such as something/things/seems before like, or as well / very well / how well, that signal a non-DM reading.
• I5. When well is a DM, it can mark rejection of a request, disagreement with a prior utterance, hesitation, or turn-taking. Detecting it therefore also signals the speaker's stance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Add per-word DM detectors for like, well and similar items, running before the tagger, so that discourse uses are not mistagged as prepositions or adverbs.
• I2 (learn): Report κ next to accuracy and compare against measured agreement between witness parsers or annotators, so we can tell when further training is pointless and labels need adjudication.
• I3 (learn): Include one-feature or majority-class classifiers as negative controls in every evaluation of a new component.
• I4 (learn): Mine such collocation cues from silver data and turn them into explicit, explainable repair rules, instead of opaque weights.
• I5 (frames): Once a token is classified as a DM, use it as a cue for speech-act labels such as disagreement or hedging, and treat DM like as a marker of approximation or reported speech.