Elamur Research⌕
☰
home / papers / 2026 / ExpressivityBench: Can LLMs Communicate Implicitly?

ExpressivityBench: Can LLMs Communicate Implicitly?

Paper · 2026 · Joshua Tint, Som Sagar, Aditya Taparia et al.

Summary

LLM benchmarks usually test explicit skills such as summarization or reasoning. This paper asks whether models can also convey information implicitly, through tone, identity cues or intent, without stating it outright. The authors build ExpressivityBench around an encoder–channel–decoder view of communication. A generator model writes text in a given domain (e.g., a song or recipe) that must carry a target signal without naming it. A blind grader then picks which signal from the category is present. Outputs that leak the signal are regenerated. Expressivity is scored as the mutual information between intended and guessed signals, plus a version normalized by label entropy. Tasks and the LLM grader were chosen through a human study, and human-written texts serve as a baseline. Across nine tasks, models do very well on affective signals like emotion but trail humans on sociolinguistic ones such as register, skill level and political slant. The work offers a reproducible way to measure implicit, human-like communication, relevant to education, mental-health support and socially aware dialogue.

Key points

  • The nine tasks cover emotions, tones, age, gender, political slant, register, skill level, genre and MPAA rating. All are in English.
  • The models evaluated are Gemma 2, Gemma 3, GPT-4.1, GPT-4o, LLaMA 2, LLaMA 3.1 and Mistral, each compared against a human-written baseline.
  • The metrics are mutual information I(s; ŝ) over the grader's confusion distribution and its normalized form N = I/H(s), which ranges from 0 (no signal transfer) to 1 (perfect transfer).
  • On the emotion task, GPT-4o and Mistral scored over three times the human baseline. The authors flag possible "hyper-expressivity" that may make text read as unnatural.
  • Performance trades off across tasks: the three best models on register (LLaMA 2, LLaMA 3.1, GPT-4.1) were the three weakest on emotions. Gemma models were among the least expressive overall, especially on gender and political slant.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Arizona State UniversityTempe · United States50 works · 2,032 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Joshua Tint, Som Sagar, Aditya Taparia, Kelly Raines, Bimsara Pathiraja, Caleb Liu and Ransalu Senanayake. 2026. ExpressivityBench: Can LLMs Communicate Implicitly?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.235

@inproceedings{tint2026expressivitybench,
  title = {{ExpressivityBench: Can LLMs Communicate Implicitly?}},
  author = {Joshua Tint, Som Sagar, Aditya Taparia, Kelly Raines, Bimsara Pathiraja, Caleb Liu and Ransalu Senanayake},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.235},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.235}
}

References

57 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gisle Andersen. 2001. Pragmatic markers and sociolinguistic variation. find
  2. Margaret Apresyan. 2018. On the concept of “expressiveness” in modern linguistics. Annals of Language and Literature. find
  3. Sameed Atif. 2023. English sentence tone analysis. https://www.kaggle.com/datasets/ sameedatif/tone-analysis. ↗
  4. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, and 1 others. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36. find
  5. Simone Benedetto, Véronique Drai-Zerbib, Marco Pedrotti, Geoffrey Tissier, and Thierry Baccino. 2013. E-readers and visual fatigue. PloS one, 8(12):e83676. find
  6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4.
  8. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Associat…
  9. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3).
  10. Mikołaj Deckert and Krzysztof Kosecki. 2023. Various dimensions of expressivity. Lodz Papers in Pragmatics, 19(1):1–6. find
  11. Xiang Chen and Xiaojun Wan. 2024. Evaluating, understanding, and improving constrained text generation for large language models. Preprint, arXiv:2310.16343. — [arXiv:2310.16343](https://arxiv.org/abs/2310.16343) find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  13. Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788. find
  14. Yifan Ding, Matthew Facciani, Amrit Poudel, Ellen Joyce, Salvador Aguinaga, Balaji Veeramani, Sanmitra Bhattacharya, and Tim Weninger. 2025. Citations and trust in llm generated responses. arXiv preprint arXiv:2501.01303. — [arXiv:2501.01303](https://arxiv.org/abs/2501.01303) find
  15. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference.
  16. Mackenzie Ferguson. 2025. Openai unleashes chatgpt’s personality customization: Now ’gen z’, ’chatty’, or whatever you want! OpenTools. Last updated: 1/10/2025. find
  17. Minje Choi, Luca Maria Aiello, Krisztián Zsolt Varga, and Daniele Quercia. 2020. Ten social dimensions of conversations and relationships. In Proceedings of The Web Conference 2020, pages 1514–1525. find
  18. Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. Do LLMs understand social knowledge? evaluating the sociability of large language models with SocKET benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11370–11403, Singa… find
  19. Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, and 1 others. 2023. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.01694. find
  20. Cristina Garbacea and Qiaozhu Mei. 2025. Why is constrained neural language generation particularly challenging? Preprint, arXiv:2206.05395. — [arXiv:2206.05395](https://arxiv.org/abs/2206.05395) find
  21. Jonathon Green. 2016. Slang: A very short introduction, volume 465. Oxford University Press. find
  22. Wm Miles Greene. 2023. Teaching tone, mood and purpose through the interpretation of acti ist poetr. find
  23. Yuling Gu, Yao Fu, Valentina Pyatkin, Ian Magnusson, Bhavana Dalvi Mishra, and Peter Clark. 2022. Just-DREAM-about-it: Figurative language understanding with DREAM-FLUTE. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 84–93, Abu Dhabi, United Arab Emirates (Hybrid)… find
  24. Ernst-August Gutt. 1996. Implicit information in literary translation: A relevance-theoretic perspective. Target. International Journal of Translation Studies, 8(2):239–256. find
  25. Scott A Crossley, Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mehrnoush Karimi, and Agnes Malatinszky. 2021. The commonlit ease of readability (clear) corpus. In EDM. find
  26. Fabian Haak and Philipp Schaer. 2023. Qbias-a dataset on media bias in search queries and query suggestions. In Proceedings of the 15th ACM Web Science Conference 2023, pages 239–244. find
  27. Scott Andrew Crossley, Perpetual Baffour, L Burleigh, and Jules King. A large-scale corpus for assessing source-based writing quality: Asap 2.0. Available at SSRN 5129353. Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, and 1… — [arXiv:2404.05221](https://arxiv.org/abs/2404.05221) find
  28. Miguel Cuevas-Alonso and Carla Míguez-Álvarez. 2021. Metadiscursive markers and text genre: A metareview. Publications, 9(4):56. find
  29. Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gašević, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE. find
  30. Adam Davies, Jize Jiang, and ChengXiang Zhai. 2023. Competence-based analysis of language models. arXiv preprint arXiv:2303.00333. — [arXiv:2303.00333](https://arxiv.org/abs/2303.00333) find
  31. Guimin Hu, Yi Xin, Weimin Lyu, Haojian Huang, Chang Sun, Zhihong Zhu, Lin Gui, Ruichu Cai, Erik Cambria, and Hasti Seifi. 2024. Recent trends of multimodal affective computing: A survey from nlp perspective. arXiv preprint arXiv:2409.07388. — [arXiv:2409.07388](https://arxiv.org/abs/2409.07388) find
  32. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa…
  33. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, and 1 others. 2024. Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning, pages 20166–20270. PMLR.
  34. Ross A. Knepper, Christoforos I. Mavrogiannis, Julia Proft, and Claire Liang. 2017. Implicit communication in a joint action. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’17, page 283–292, New York, NY, USA. Association for Computing Machinery. find
  35. Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. 2024. A systematic survey and critical review on evaluating large language mode…
  36. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. find
  37. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federati…
  38. Yufei Li, John Nham, Ganesh Jawahar, Lei Shu, David Uthus, Yun-Hsuan Sung, Chengrun Yang, Itai Rolnick, Yi Qiao, and Cong Liu. 2025. Dr genre: Reinforcement learning from decoupled llm feedback for generic text rewriting. arXiv preprint arXiv:2503.06781. find
  39. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.
  40. Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024. " we need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems,… find
  41. Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, and Barbara Plank. 2025. Pragmatics in the era of large language models: A survey on datasets, evaluation, opportunities and challenges. Preprint, arXiv:2502.12378.
  42. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  43. ArXiv, Larysa Ivanivna Orekhova, Tetiana Gremaliuk, Natalia Borysenko, and Oksana Cheban. 2022. Linguistic and stylistic markers of influence in the essayistic text: A linguophilosophic aspect. find
  44. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models.
  45. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the association for computational linguistics, 4:61–74.
  46. Flor Miriam Plaza-del Arco, Alba A. Cercas Curry, Amanda Cercas Curry, and Dirk Hovy. 2024. Emotion analysis in NLP: Trends, gaps and roadmap for future directions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING… find
  47. Shireesh Reddy Pyreddy and Tarannum Shaila Zaman. 2025. Emoxpt: Analyzing emotional variances in human comments and llm-generated responses. arXiv preprint arXiv:2501.06597. — [arXiv:2501.06597](https://arxiv.org/abs/2501.06597) find
  48. Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  49. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Com… find
  50. Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, and David West Brown. 2025. Do llms write like humans? variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(8). find
  51. Catherine Anderson; Bronwyn Bjorkman; Derek Denis; Julianne Doner; Margaret Grant; Nathan Sanders; and Ai Taniguchi. 2022. Essentials of Linguistics, chapter 7. eCampusOntario. find
  52. Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James Pennebaker. 2006. Effects of age and gender on blogging. In Proceedings of the 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs. find
  53. Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. 2024. A critical evaluation of ai feedback for aligning large language models. arXiv preprint arXiv:2402.12366. — [arXiv:2402.12366](https://arxiv.org/abs/2402.12366) find
  54. Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. Preprint, arXiv:2401.07078.
  55. PS Sreeja and GS Mahalakshmi. 2019. Perc-an emotion recognition corpus for cognitive poems. In 2019 International Conference on Communication and Signal Processing (ICCSP), pages 0200–0207. find
  56. IEEE. 2024, pages 2004–2018, St. Julian’s, Malta. Association for Computational Linguistics. find
  57. Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core competency: Survey on evaluation of large language models. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Fr…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tint-2026-expressivitybench-can-llms-communicate-implicitly/