📄 How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction

Summary

The paper asks how well language models and people can recover the English discourse marker actually when it is removed from real conversational speech, and how much surrounding talk helps. The authors took 295 adult utterances containing actually from the Providence Corpus (PhonBank), which records home interactions around young children. They built cloze items in four context conditions: target alone, target plus next utterance, previous utterance plus target, and all three. Answers were open-ended, not chosen from a list. mTurk workers and five models (BERT, RoBERTa, ELECTRA, GPT-3.5, GPT-4) each produced five guesses, scored by recall@5. Context always helped, but how much, and from which side, varied by reader. Preceding context mattered most for humans and GPT-4, and GPT-4 slightly beat humans overall. On GPT-3.5-generated filler items, several models far outscored humans. This suggests synthetic text favours models and can distort human–model comparisons, so pragmatic evaluation needs natural data and careful context design.

Key points

• Data: 295 natural actually utterances from the Providence Corpus, each with one preceding and one following utterance. About 70% are child-directed. The set adds 37 corpus distractors and 295 GPT-3.5 synthetic items, for 2360 cloze items across four conditions (T, T+N, P+T, P+T+N).
• Humans: 255 mTurk participants, about 8.1 items each, each giving 5 candidate words. Responses with many repeated answers or words copied from the prompt were filtered out.
• On actually items, the best recall@5 scores came with full context: GPT-4 34.2, humans 31.5, RoBERTa 25.8, GPT-3.5 22.4, BERT 21.0, ELECTRA 13.6.
• Adding the preceding utterance gave GPT-4 roughly +8 points and GPT-3.5 roughly +6 points. RoBERTa gained little from context, yet it beat GPT-3.5 in every condition.
• On synthetic items with full context, GPT-4 reached 91.2 versus 51.2 for humans. Every model except ELECTRA beat humans there. A post-hoc check found that people who did more items guessed actually somewhat more often, but the effect was moderate.

Interesting ideas

• I1. Testing the same item under target-only, left-context, right-context and both-sides conditions separates how much each side of the discourse contributes, rather than giving one blended score.
• I2. The authors let readers produce answers freely instead of picking from a fixed list, because a closed list makes the task easier and inflates scores.
• I3. Text written by a model is unusually predictable for models, so a model-generated test set measures fit to that model's distribution more than real competence.
• I4. Discourse markers like actually signal contrast and expectation through regular cues in nearby utterances, which makes them predictable from the surrounding conversation.
• I5. The authors used filler items aimed at other words to keep annotators from learning the hidden target, then checked afterwards whether the target-answer rate rose with exposure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Run the prag implicature and speech-act components with and without the previous and next sentence, to measure which pragmatic decisions truly depend on prior discourse.
• I2 (answer): Score the fable and tale QA readers on unconstrained answers as well as multiple choice, so that guessing by elimination cannot hide weak understanding.
• I3 (learn): Keep a natural-text evaluation split apart from any LLM-teacher-generated data, and report the gap between the two as a negative control.
• I4 (frames): Treat actually and similar markers as cues that a statement contradicts an earlier one or an expected belief, and link them in prag to the frame they correct.
• I5 (learn): When humans or LLM teachers label targeted phenomena, mix in decoy items and check for annotator drift before accepting the silver data.