📄 Towards Automatic Identification of Discourse Markers in Dialogs: The Case of Like

Summary

The paper addresses a first step in pragmatic dialog analysis: deciding when an ambiguous word is a discourse marker (DM) rather than an ordinary lexical item. The focus is like, which can also be a verb, preposition, conjunction, adjective, noun or adverb. As a DM, it mostly signals that what follows is a loose approximation. Using transcribed staff meetings from the ICSI Meeting Recording corpus, the authors first measure how well human judges agree, comparing transcripts alone with transcripts plus audio. They then build automatic detectors from three kinds of evidence: collocation filters, POS tags, and timing features such as pause and word duration. These are combined with C4.5 decision trees in WEKA. Humans reach reliable agreement, around κ ≈ 0.74, only when they can hear the recordings. Automatically, collocation filters dominate: about 70% precision at near-full recall for like, and about 91% precision for well. Off-the-shelf POS taggers fail on DMs. The results suggest high-recall filters can pre-screen data for annotators and support later parsing and tagging.

Key points

• Data: ICSI-MR corpus, made of 75 one-hour meetings with up to eight speakers each. The authors use an initial release of 50 dialogs, segmented into roughly 65,000 prosodic utterances. In about 50 hours, 37% of 2,116 occurrences of like are DMs.
• Human annotation:
  • The first experiment used 6 judges (native English and French speakers) working from transcripts only.
  • The second used 3 judges who could also listen to the audio.
  • With sound, agreement reached κ ≈ 0.74.
• For like, a tree using only the collocation filters gets 0.96 recall and 0.67 precision (80% correct, κ = 0.62), close to the best tree. Without collocations, timing features alone give 67% correct with κ = 0.23.
• Timing cue: a pause longer than 240 ms before like marked a DM in 70 of 78 remaining cases. Very short tokens (under 120 ms) tended not to be DMs.
• For well, 579 of 873 occurrences are DMs (a 66% baseline). Collocations plus the pause after well give 91% accuracy, κ = 0.8, 91% precision and 97% recall.

Interesting ideas

• I1. Measure human agreement under different information conditions (text only vs. text plus audio) before building a classifier, so that the automatic score has a realistic ceiling.
• I2. Use cheap lexical-context collocation filters that rule out non-DM readings (e.g. as well, I like) as a high-recall first pass, and leave the rest to finer analysis or humans.
• I3. Also add positive filters that confirm a DM (e.g. oh well, OK well, a word ending an abandoned utterance), alongside the negative ones.
• I4. Treat DM like as a loose-talk signal: what follows is an approximation, whether of a quantity or of a wording.
• I5. Features that generalize across a whole class of markers, such as timing, are distinguished from item-specific collocations that win for individual words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluate pragmatic labels against agreement measured when annotators see only the text, the same input Mova has, rather than an idealized gold standard.
• I2 (tokens): Add explainable pre-tagging rules that flag possible DM readings of like/well/so, so that the tagger does not force them into verb or preposition tags.
• I3 (tree): Detect such markers before parsing and attach them as discourse dependents, so they do not distort the rest of the UD tree.
• I4 (frames): When like precedes a number or an NP, mark the value or description as approximate, so the frame carries hedged rather than exact content.
• I5 (world): Store hedged quantities from like twenty minutes as ranges, not point values, so that later reasoning about measures and event durations stays honest.