Elamur Research⌕
☰
home / papers / 2023 / Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective

Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective

Paper · 2023 · Khanh Nguyen

Summary

The paper asks what kind of "model of thought" large language models implement. It offers a conceptual and mathematical answer rather than new data. The author introduces the bounded pragmatic speaker (BPS), a generalisation of the Rational Speech Act model. A speaker with limited computation combines two modules. A base speaker is a prior that shrinks the space of candidate utterances. A theory-of-mind listener estimates how a real listener would decode an utterance. The output distribution is the product of the two, a Bayesian update that is found through approximate inference.

The paper's main findings are these:

  • A plain language model is a degenerate BPS in which one network plays both roles.
  • Re-rankers and RLHF reward models add a separate, stronger listener.
  • RLHF fine-tuning can be read as variational approximation of the BPS distribution. On this reading, the reward model acts as a slow, deliberate system and the tuned policy acts as a fast, heuristic one, echoing Kahneman's dual-process picture.

The framing matters because it makes LLMs modular and diagnosable. It also points to concrete next steps: world-model-like rewards and feedback richer than scalar scores.

Key points

  • The BPS speaking distribution is S_bps(u | z, c) ∝ S_base(u | z, c) · L_ToM(z | u, c). The base speaker is the prior and the ToM listener is the likelihood. An "unbounded" speaker would instead score every utterance with the true listener L_real.
  • The paper names three failure sources, each pointing to an improvement direction:
    • a weak base speaker (search capability), improved by training on more data;
    • an inaccurate ToM listener (pragmatic capability), improved by re-rankers or RLHF reward models;
    • a poor inference algorithm, improved by better decoding methods.
  • To diagnose which module is at fault, the paper cites Zhao et al. (2023a): compare a model against an oracle that matches it in the capability being tested.
  • The paper identifies two limits of RLHF:
    • Reward functions trained only on text and human judgements lack real world knowledge. The proposed remedy is to generalise the reward into a world model.
    • Variational inference passes only a score back to the policy, which makes learning sample-inefficient. Structured or language feedback is suggested instead, citing Nguyen et al. (2021).
  • The paper is a position/theory paper. It appeared at ICML 2023's inaugural workshop dedicated to Theory of Mind in Communicating Agents and has no experiments of its own.

Interesting ideas

  • I1. Generation can be split into a cheap proposer that keeps only plausible candidates and a separate listener model that predicts which candidate will be understood as intended. Their product picks the output, and each part can be upgraded on its own.
  • I2. To find which module causes a failure, swap one module for an oracle that is perfect at that single capability and measure how much the end task improves. This avoids spreading effort across every possible fix.
  • I3. A scalar reward forces the learner to guess the target's shape from many samples. Returning structured information about the hidden intention lets learning generalise across overlapping, compositional cases.
  • I4. The judge that scores outputs should reason about consequences in a model of the world, not just surface plausibility learned from text.
  • I5. When the hidden intention and the output share one representation, rich feedback collapses into plain demonstrations, i.e. behaviour cloning. The benefit therefore comes from feedback living in a different, structured space.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Let the generator propose several surface realisations of a meaning graph. Then run each one back through Mova's own parser and frame extractor as the "listener", and keep the realisation whose recovered graph best matches the source graph.
  • I2 (learn): In evaluation, substitute gold UD trees or gold frames one stage at a time on school-test and fable QA. Attribute answer errors to the parser, frames, or world stage before deciding what to retrain.
  • I3 (learn): Ask LLM teacher tiers for structured corrections instead of accept/reject verdicts, such as the wrong arc and its right head or the missing semantic role. Self-induced repair rules could then be learned from far fewer examples.
  • I4 (answer): Check candidate answers against the world of the text (entities, holdings, event order) before emitting them. Treat that simulation as the "slow" verifier behind a faster answer heuristic.
  • I5 (learn): Design teacher feedback for silver data in terms of frames or meaning graphs rather than raw corrected sentences, so it carries structure beyond copying.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States113 works · 48,612 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksPsycholinguistics & language acquisition · 108 works

Cite

Cite as Khanh Nguyen. 2023. Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective. arXiv:2305.17760.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.17760 · arXiv 2305.17760

@misc{nguyen2023language,
  title = {{Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective}},
  author = {Khanh Nguyen},
  year = {2023},
  doi = {10.48550/arxiv.2305.17760},
  eprint = {2305.17760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.17760}
}

References

58 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. ↗
  2. Andreas, J. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769–5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  3. Andreas, J. and Klein, D. Reasoning about pragmatics with neural listeners and speakers. arXiv preprint arXiv:1604.00562, 2016. ↗
  4. Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
  5. Baron-Cohen, S., Leslie, A. M., and Frith, U. Does the autistic child have a “theory of mind”? Cognition, 21(1): 37–46, 1985. find
  6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  7. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. ↗
  8. Chiu, S.-H. and Chen, B. Innovative bert-based reranking language models for speech recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 266– 271. find
  9. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. ↗
  10. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  11. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. ↗
  12. Corona, R., Fried, D., Devin, C., Klein, D., and Darrell, T. Modular networks for compositional instruction following. arXiv preprint arXiv:2010.12764, 2020. ↗
  13. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022. ↗
  14. Frank, M. C. and Goodman, N. D. Predicting pragmatic reasoning in language games. Science, 336(6084):998– 998, 2012.
  15. Fried, D., Andreas, J., and Klein, D. Unified pragmatic models for generating and following instructions. arXiv preprint arXiv:1711.04987, 2017. ↗
  16. Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. find
  17. Gershman, S. and Goodman, N. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science society, volume 36, 2014. find
  18. Goodman, N. D. and Frank, M. C. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829, 2016. find
  19. Gopnik, A. and Astington, J. W. Children’s understanding of representational change and its relation to the understanding of false belief and the appearance-reality distinction. Child development, pp. 26–37, 1988. find
  20. Griffiths, T. L., Chater, N., Kemp, C., Perfors, A., and Tenenbaum, J. B. Probabilistic models of cognition: Exploring representations and inductive biases. Trends in cognitive sciences, 14(8):357–364, 2010. find
  21. Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. ↗
  22. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. ↗
  23. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. ↗
  24. Kahneman, D. Thinking, fast and slow. macmillan, 2011. find
  25. Korbak, T., Perez, E., and Buckley, C. L. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022. ↗
  26. Lampinen, A. K., Chan, S. C., Dasgupta, I., Nam, A. J., and Wang, J. X. Passive learning of active causal strategies in agents and language models. arXiv preprint arXiv:2305.16183, 2023. ↗
  27. Lazaridou, A., Peysakhovich, A., and Baroni, M. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016. ↗
  28. Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018. ↗
  29. Lewis, D. K. Convention: A Philosophical Study. Cambridge, MA, USA: Wiley-Blackwell, 1969. find
  30. Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022. ↗
  31. Lu, X., Welleck, S., West, P., Jiang, L., Kasai, J., Khashabi, D., Bras, R. L., Qin, L., Yu, Y., Zellers, R., et al. Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. arXiv preprint arXiv:2112.08726, 2021. ↗
  32. Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, E. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023. ↗
  33. Nguyen, K. X., Misra, D., Schapire, R., Dudı́k, M., and Shafto, P. Interactive learning from activity description. In International Conference on Machine Learning, pp. 8096–8108. PMLR, 2021. find
  34. Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-t., Wang, S. I., and Lin, X. V. Lever: Learning to verify language-to-code generation with execution. arXiv preprint arXiv:2302.08468, 2023. ↗
  35. OpenAI. Chatgpt. https://openai.com/blog/chatgpt, 2022. ↗
  36. OpenAI. Gpt-4 technical report. 2023.
  37. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  38. Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. ↗
  39. Premack, D. and Woodruff, G. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4): 515–526, 1978. find
  40. Sanborn, A. N. and Chater, N. Bayesian brains without probabilities. Trends in cognitive sciences, 20(12):883– 893, 2016. find
  41. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. ↗
  42. Simon, H. A. Models of man; social and rational. 1957. find
  43. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021, 2020. find
  44. Sumers, T., Hawkins, R., Ho, M. K., Griffiths, T., and Hadfield-Menell, D. How to talk so ai will learn: Instructions, descriptions, and autonomy. Advances in Neural Information Processing Systems, 35:34762–34775, 2022. find
  45. Tenenbaum, J. B., Kemp, C., Griffiths, T. L., and Goodman, N. D. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011. find
  46. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. ↗
  47. Vul, E., Goodman, N., Griffiths, T. L., and Tenenbaum, J. B. One and done? optimal decisions from very few samples. Cognitive science, 38(4):599–637, 2014. find
  48. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. 2023. find
  49. Wang, R. E., White, J., Mu, J., and Goodman, N. D. Calibrate your listeners! robust communication-based training for pragmatic speakers. arXiv preprint arXiv:2110.05422, 2021. ↗
  50. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. ↗
  51. White, J., Mu, J., and Goodman, N. D. Learning to refer informatively by amortizing pragmatic reasoning. arXiv preprint arXiv:2006.00418, 2020. ↗
  52. Wimmer, H. and Perner, J. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 13(1):103–128, 1983. find
  53. Wong, L. S., Grand, G., Lew, A. K., Goodman, N. D., Mansinghka, V. K., Andreas, J., and Tenenbaum, J. B. From word models to world models: Translating from natural language to the probabilistic language of thought. 2023. find
  54. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. ↗
  55. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022a. ↗
  56. Zhang, T., Yu, T., Hashimoto, T. B., Lewis, M., Yih, W.-t., Fried, D., and Wang, S. I. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211.16490, 2022b. ↗
  57. Zhao, L., Nguyen, K., and Daumé III, H. Define, evaluate, and improve task-oriented cognitive capabilities for instruction generation models. arXiv preprint arXiv:2301.05149, 2023a. ↗
  58. Zhao, Z., Lee, W. S., and Hsu, D. Large language models as commonsense knowledge for large-scale task planning. arXiv preprint arXiv:2305.14078, 2023b. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2023-language-models-bounded-pragmatic-speakers/