Elamur Research⌕
☰
home / papers / 2026 / Rhetorical Questions in LLM Representations: A Linear Probing Study

Rhetorical Questions in LLM Representations: A Linear Probing Study

Summary

The paper asks how large language models internally encode rhetorical questions, which are asked to persuade or express stance rather than to obtain information. Earlier work mostly treated this as a labeled classification or QA problem. The authors instead apply linear probes to frozen hidden states. They use two social-media corpora: RQ (Twitter, a question plus the preceding tweet) and SRAQ (Reddit comments). Last-token or mean-pooled representations are reduced to a 64-dimensional PCA space and analyzed with a diffMean direction, logistic regression and a linear SVM.

Within each dataset, rhetorical and information-seeking questions are linearly separable. The signal appears in early layers and is most stable for last-token states. Probes still work when transferred across datasets. However, directions learned from different corpora rank the same examples quite differently. Qualitative inspection links these differences to distinct phenomena: the SRAQ direction favors stance embedded in longer argumentation, while the RQ direction favors short, form-driven questions.

The main lesson is that high probe accuracy or successful transfer does not show that a single shared feature exists. Rhetoricity appears to be spread over several context-dependent directions.

Key points

  • Datasets:
    • RQ (Zhuang and Riloff, 2020) has 4,997 tweets, of which 2,332 are rhetorical, split 3,200/797/1,000. Inputs with context average 38.9 tokens.
    • SRAQ (Ikumariegbe et al., 2025) has 971 Reddit questions, of which 609 are rhetorical, split 384/103/484. The Paragraph formulation averages 58.5 tokens, and Full_turn results are in Appendix G.
  • Transfer and agreement: cross-dataset transfer reaches AUROC of roughly 0.7–0.8. Yet the overlap between top-ranked instances from different probes is often below 0.2.
  • Length contrast: examples ranked highest by the in-domain SRAQ diffMean direction are longer than those ranked highest by the transferred RQ direction. The averages are 188.8 vs. 126.4 tokens for the top 1% and 150.9 vs. 116.4 tokens for the top 3%, counted with the Llama-3.3-70B-Instruct tokenizer.
  • Misranking by form: the RQ direction sometimes places gold-informational questions near the top because of their interrogative surface form.
  • Scope and code: the analysis is linear-only. Preliminary steering experiments hint at controllability but are left for future causal work. Code is available at github.com/ruyi101/rq-representation-probing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.5

@inproceedings{yao2026rhetorical,
  title = {{Rhetorical Questions in LLM Representations: A Linear Probing Study}},
  author = {Yao, Louie Hong and Anand, Vishesh and Zhuang, Yuan and Jiang, Tianyu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.5},
  url = {https://doi.org/10.18653/v1/2026.acl-long.5}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, and others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. ↗ our card
  2. Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. ↗ our card
  3. Shohini Bhattasali, Jeremy Cytryn, Elana Feldman, and Joonsuk Park. 2015. Automatic identification of rhetorical questions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015). find
  4. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations (ICLR 2024). ↗
  5. Lucy Farnik, Tim Lawson, Conor Houghton, and Laurence Aitchison. 2025. Jacobian sparse autoencoders: Sparsify computations, not just activations. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). ↗
  6. Jane Frank. 1990. You call that a rhetorical question?: Forms and functions of rhetorical questions in conversation. Journal of Pragmatics, 14:723–738. ↗
  7. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations (ICLR 2025). ↗
  8. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024). ↗
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  10. Chung-hye Han. 2002. Interpreting interrogatives as rhetorical questions. Lingua, 112:201–229. ↗
  11. Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. 2025. Sparse autoencoders can interpret randomly initialized transformers. arXiv preprint arXiv:2501.17727. ↗
  12. Eghbal A. Hosseini and Evelina Fedorenko. 2023. Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS 2023). ↗
  13. Oghenevovwe Ikumariegbe, Eduardo Blanco, and Ellen Riloff. 2025. Studying rhetorically ambiguous questions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025). find
  14. Shiyu Ji, Farnoosh Hashemi, Joice Chen, Juanwen Pan, Weicheng Ma, Hefan Zhang, Sophia Pan, Ming Cheng, Shubham Mohole, Saeed Hassanpour, Soroush Vosoughi, and Michael Macy. 2025. A generalizable rhetorical strategy annotation model using LLM-based debate simulation and labelling. In Findings of the Association for Computational Linguistics: EMNLP 2025 (Findings of EMNLP 2025). ↗
  15. Daniel Jurafsky, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1997. Automatic detection of discourse structure for speech recognition and understanding. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 88–95. IEEE. ↗
  16. Zlata Kikteva, Alexander Trautsch, Steffen Herbold, and Annette Hautli-Janisz. 2024. Question type prediction in natural debate. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL 2024). find
  17. Joshua Lee, Wyatt Fong, Alexander Le, Sur Shah, Kevin Han, and Kevin Zhu. 2025. Pragmatic metacognitive prompting improves LLM performance on sarcasm detection. In Proceedings of the 1st Workshop on Computational Humor (CHum). ↗
  18. Daniel Vennemeyer, Phan Anh Duong, Tiffany Zhan, and Tianyu Jiang. 2025. Sycophancy is not one thing: Causal separation of sycophantic behaviors in llms. arXiv preprint arXiv:2509.21305. ↗
  19. Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling (COLM 2024). ↗ our card
  20. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022). ↗
  21. Shereen Oraby, Vrindavan Harrison, Amita Misra, Ellen Riloff, and Marilyn Walker. 2017. Are you serious?: Rhetorical questions and sarcasm in social media dialog. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL 2017). find our card
  22. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. ↗ our card
  23. Jingyi Qiu, Hong Chen, and Zongyi Li. 2025. Counterfactual llm-based framework for measuring rhetorical style. arXiv preprint arXiv:2512.19908. ↗
  24. Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, and David West Brown. 2025. Do llms write like humans? variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(8):e2422455122. ↗
  25. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, and 1 others. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). find our card
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. ↗ our card
  27. Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari, and Jing Qin. 2025. Sarcasmbench: Towards evaluating large language models on sarcasm understanding. IEEE Transactions on Affective Computing, 16(4):2560–2578. ↗
  28. Yuan Zhuang and Ellen Riloff. 2020. Exploring the role of context to distinguish rhetorical and information-seeking questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. find
  29. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). ↗
  30. Richard M Roberts and Roger J Kreuz. 1994. Why do people use figurative language? Psychological science, 5:159–163. ↗
  31. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by layer: Uncovering hidden representations in language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). ↗
  32. Džemal Špago. 2016. Rhetorical questions or rhetorical uses of questions? Explorations in English Language and Linguistics, 4:102–115. ↗
  33. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248. Effect of PCA Truncation on Linear Probing ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/yao-2026-rhetorical-questions-llm-representations-linear/