Elamur Research⌕
☰
home / papers / 2024 / Training LLMs to Recognize Hedges in Spontaneous Narratives

Training LLMs to Recognize Hedges in Spontaneous Narratives

Paper · 2024 · Amie J. Paige, Adil Soubki, John Murzaku et al.

Summary

Hedges are markers like "sort of", "like" or "I think". Speakers use them to flag what they say as tentative, approximate, or coming from someone else. The authors test whether current language models can find these markers in transcribed spontaneous speech, as a first step toward systems that handle such collateral signals in conversation. They used 63 retellings of Roadrunner cartoons, told from memory by 21 speakers to listeners in the same room (Galati and Brennan, 2010). Human coders labelled hedges token by token, producing the Roadrunner-Hedge corpus. Three approaches were compared: fine-tuning BERT as a token classifier, and zero- and few-shot prompting of GPT-4o and LLaMA-3. Fine-tuned BERT won clearly (F1 about 0.91), with few-shot GPT-4o second. An error analysis of the strongest systems then guided an LLM-in-the-Loop revision of the gold labels. This raised scores for every model and exposed genuinely ambiguous cases. The result matters because a small supervised encoder still beats large prompted models on a subtle pragmatic tagging task.

Key points

  • Data: 63 Roadrunner cartoon narratives from 21 speakers, transcribed from face-to-face sessions. The annotated resource is the Roadrunner-Hedge corpus, with code at github.com/cogstates/hedging.

  • The coding scheme lists hedge types, each with exclusion conditions:

    • like, except as simile, verb or comparison
    • you know, except when it reports someone's knowledge or acts as a discourse marker
    • just, except when it means "only"
    • approximators/rounders
    • proxies for details the speaker cannot or will not recall
    • hedging morpheme suffixes
    • expressions of doubt
    • tag questions and try markers

  • Detection is done per token, not per clause. This is the main contrast with the earlier work of Raphalen et al. (2022).

  • F1 on the original gold vs. the LLM-in-the-Loop corrected gold:

    ModelOriginal goldCorrected goldError reduction
    BERT0.9080.92518.5%
    GPT-4o few-shot0.7120.7213.1%
    GPT-4o zero-shot0.5100.5518.4%

  • The theoretical frame follows Prince et al. (1982):

    • approximators make content less exact;
    • shield hedges change the speaker's stance toward the content, split into plausibility shields and attribution shields.

Interesting ideas

  • I1. Use the confident disagreements between strong models and the gold labels as a worklist for re-checking annotation, then report how much error the correction removes for each model; this turns error analysis into a cheap gold-repair loop.
  • I2. Define each ambiguous hedge form with an explicit "counts unless used as X" test (e.g., just is not a hedge when it means "only"); this makes token-level labels reproducible and points to the context that disambiguates them.
  • I3. Separate approximators (loosening content) from plausibility shields (weaker commitment) and attribution shields (responsibility moved to another source); the three kinds have different consequences for what a reader should believe.
  • I4. Quotative like ("he's like whatever...") counts as a hedge because it signals a dramatized, non-verbatim rendering of someone's reaction.
  • I5. The same string can be a hedge or not depending on discourse context, e.g. "I think" answering a direct "what do you think?" question; string lookup alone is therefore insufficient.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Where witness parsers or LLM teacher tiers agree with each other but contradict silver or gold data, queue those spots for review, and measure error reduction per model as a check that the repair is real.
  • I2 (frames): Such exclusion tests can be written as rules over the UD tree, e.g. like as a verb or comparison head vs. a discourse attachment, so the hedge decision stays explainable.
  • I3 (world): Store a hedged claim as a state with a commitment level and a source, rather than as plain fact, so later reasoning can weigh it accordingly.
  • I4 (frames): Treat such constructions as reported-speech or thought frames attributed to the character, but flagged as loose paraphrase rather than an exact utterance.
  • I5 (frames): Decide hedge status from the speech-act context of the preceding turn, avoiding a lexicon-only (bag-of-words) shortcut.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States43 works · 2,571 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Amie J. Paige, Adil Soubki, John Murzaku, Owen Rambow and Susan E. Brennan. 2024. Training LLMs to Recognize Hedges in Spontaneous Narratives. SIGDIAL 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.03319 · arXiv 2408.03319

@inproceedings{paige2024training,
  title = {{Training LLMs to Recognize Hedges in Spontaneous Narratives}},
  author = {Amie J. Paige, Adil Soubki, John Murzaku, Owen Rambow and Susan E. Brennan},
  booktitle = {SIGDIAL 2024},
  year = {2024},
  doi = {10.48550/arxiv.2408.03319},
  eprint = {2408.03319},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.03319}
}

References

54 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alafate Abulimiti, Chloé Clavel, and Justine Cassell. 2023a. When to generate hedges in peer-tutoring interactions. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 572–583, Prague, Czechia. Association for Computational Linguistics. ↗
  2. Alafate Abulimiti, Chloé Clavel, and Justine Cassell. 2023b. How about kind of generating hedges using end-to-end neural models? arXiv:2306.14696. Preprint, AI@Meta. 2024. Llama 3 model card. ↗
  3. Jennifer Arnold, Maria Fagnano, and Michael Tanenhaus. 2003. Disuencies signal theee, um, new information. Journal of Psycholinguistic Research, 32:25–36. find
  4. Jennifer Arnold, Carla Hudson Kam, and Michael Tanenhaus. 2007. If you say thee uh you are describing something hard: The on-line attribution of disfluency during reference comprehension. Journal of experimental psychology. Learning, memory, and cognition, 33:914–30. ↗
  5. Susan E. Brennan. 2005. How conversation is shaped by visual and spoken evidence. In John Trueswell and Michael Tanenhaus, editors, Approaches to studying world-situated language use: Bridging the language-as-product and language-as-action traditions, pages 95–129. MIT Press. find
  6. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6):1482–1493. ↗
  7. Susan E. Brennan and J. O. Ohaeri. 1999. Why do electronic conversations seem less polite? the costs and benefits of hedging. In ACM SIGSOFT Software Engineering Notes. ↗
  8. Susan E Brennan and Maurice Williams. 1995. The feeling of another’s knowing: Prosody and filled pauses as cues to listeners about the metacognitive states of speakers. Journal of Memory and Language, 34(3):383–398. find
  9. P. Brown and S. C. Levinson. 1987. Politeness: Some universals in language usage. Cambridge University Press.
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. H. chi, jeff dean, jacob devlin, adam roberts, denny zhou, quoc v. le, and jason wei. 2022. scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. ↗
  11. Herbert H. Clark. 1994. Managing problems in speaking. Speech Communication, 15:243–250. ↗
  12. Herbert H. Clark. 1996. Using Language. “Using” Linguistic Books. Cambridge University Press. find
  13. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In L. B. Resnick, J. Levine, and S. D. Teasley, editors, Perspectives on Socially Shared Cognition, pages 127–149. American Psychological Association. ↗
  14. Herbert H. Clark and Richard J. Gerrig. 1990. Quotations as demonstrations. Language, 66:764–805. ↗
  15. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. ↗
  16. Delphine Dahan. 2023. Collaboration under uncertainty in unscripted conversations: The role of hedges. Journal of Experimental Psychology: Learning, Memory, and Cognition, 49:320–335. ↗
  17. Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. 2023. LLM-in-the-loop: Leveraging large language model for thematic analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9993–10001, Singapore. Association for Computational Linguistics. ↗
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  20. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 68–73, Suntec, Singapore. Association for Computational Linguistics. ↗
  21. Bruce Fraser. 2010. Pragmatic competence: The case of hedging. New Approaches to Hedging, 9:15–34. ↗
  22. Alexia Galati and Susan E. Brennan. 2010. Attenuating information in spoken communication: For the speaker, or for the addressee? Journal of Memory and Language, 62:35–51. ↗
  23. Alexia Galati and Susan E. Brennan. 2014. Speakers adapt gestures to addressees’ knowledge: implications for models of co-speech gesture. Language, Cognition and Neuroscience, 29:435 – 451. ↗
  24. Alexia Galati and Susan E. Brennan. 2021. What is retained about common ground? Distinct effects of linguistic and visual co-presence. Cognition, 215. ↗
  25. H Paul Grice. 1957. Meaning. The philosophical review, 66(3):377–388. find
  26. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗
  27. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  28. Nanjiang Jiang and Marie-Catherine de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails on pragmatics. Transactions of the Association for Computational Linguistics, 9:1081–1097. ↗
  29. George Lakoff. 1975. Hedges: A study in meaning citeria and the logic of fuzzy concepts. Journal of Philosophical Logic, pages 458–508. ↗
  30. Robin Lakoff. 1973. Language and woman’s place. Language in Society, 2(1):45–79. ↗
  31. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal. Association for Computational Linguistics. ↗
  32. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.
  33. Kris Liu and Jean Fox Tree. 2012. Hedges enhance memory but inhibit retelling. Psychonomic bulletin & review, 19:892–8. ↗
  34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  35. Michael A. Madaio, Justine Cassell, and Amy E. Ogan. 2017. “i think you just got mixed up”: confident peer tutors hedge to support partners’ face needs. International Journal of Computer-Supported Collaborative Learning, 12:401–421. ↗
  36. John Murzaku, Tyler Osborne, Amittai Aviram, and Owen Rambow. 2023. Towards generative event factuality prediction. In Findings of the Association for Computational Linguistics: ACL 2023, pages 701– 715, Toronto, Canada. Association for Computational Linguistics. ↗
  37. John Murzaku, Adil Soubki, and Owen Rambow. 2024. Multimodal belief prediction. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2024. International Speech Communication Association. ↗
  38. John Murzaku, Peter Zeng, Magdalena Markowska, and Owen Rambow. 2022. Re-examining FactBank: Predicting the author’s presentation of factuality. In Proceedings of the 29th International Conference on Computational Linguistics, pages 786–796, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  39. OpenAI. 2024. Gpt-4o. index/hello-gpt-4o/. https://openai.com/ ↗
  40. Amanda Pogue and Michael K. Tanenhaus. 2018. Learning from uncertainty: exploring and manipulating the role of uncertainty on expression production and interpretation. In Proceedings of the 40th Annual Conference of the Cognitive Science Society. find
  41. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4393–4399, Florence, Italy. Association for Computational Linguistics. ↗
  42. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014–1022, Beijing, China. Coling 2010 Organizing Committee.
  43. E. F. Prince, J. Frader, and C. Bosk. 1982. On hedging in physician-physician discourse. In Robert Di Prieto, editor, Linguistics and the Professions, pages 83–97. Albex. find
  44. Anna Prokofieva and Julia Hirschberg. 2014. Hedging and speaker commitment. In 5th International Workshop on Emotion, Social Signals, Sentiment & Linked Open Data. find
  45. Yann Raphalen, Chloé Clavel, and Justine Cassell. 2022. “You might think about slightly revising the title”: Identifying hedges in peer-tutoring interactions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2174, Dublin, Ireland. Association for Computational Linguistics. ↗
  46. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  47. Mark Sabbagh and Dare Baldwin. 2001. Learning words from knowledgeable versus ignorant speakers: Links between preschoolers’ theory of mind and semantic development. Child Development, 72:1054 – 1070. ↗
  48. Roser Saurí and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. Language resources and evaluation, 43:227–268. ↗
  49. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. Preprint, arXiv:2311.09718. ↗
  50. Vicki L. Smith and Herbert H. Clark. 1993. On the course of answering questions. Journal of Memory and Language, 32(1):25–38. ↗
  51. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357, Vancouver, Canada. Association for Computational Linguistics. ↗
  52. Morgan Ulinski, Seth Benjamin, and Julia Hirschberg. 2018. Using hedge detection to improve committed belief tagging. In Proceedings of the Workshop on Computational Semantics beyond Events and Roles, pages 1–5, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  53. Morgan Ulinski and Julia Hirschberg. 2019. Crowdsourced hedge term disambiguation. In LAW@ACL. ↗
  54. Jiarui Yao, Haoling Qiu, Jin Zhao, Bonan Min, and Nianwen Xue. 2021. Factuality assessment as modal dependency parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1540–1550, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paige-2024-training-llms-recognize-hedges-spontaneous/