Elamur Research
☰
home / papers / 2025 / Speech Act Patterns for Improving Generalizability of Explainable Politeness Detection Models

Speech Act Patterns for Improving Generalizability of Explainable Politeness Detection Models

What it is about

Explainable politeness classifiers built on word features do well in the domain they were trained on and badly elsewhere. Here GPT-4 marks, inside each request of the Stanford Politeness Corpus, the words, phrases or sentences that perform one of ten acts derived from the Danescu-Niculescu-Mizil politeness strategies (gratitude, greeting, apology, direct question, direct start, hedging, positive and negative sentiment, request, deference); a manual check of 100 marked spans found 83% right, with hedging and direct questions confused most. The spans of each act are clustered by contextual embeddings with a label-aware decision tree, so «I am sorry» and «apologies» land together, and the cluster ids become features of a logistic regression or SVM. Trained on Wikipedia requests and tested on Stack Exchange, act clusters plus words reach 71.3% against 67.2% for the best earlier explainable model (fine-tuned BERT 75.5%); in-domain they do slightly worse than word clusters.

Ideas

For Mova

languages/en/32-politeness-and-hedging.md §6 (computational politeness; Mova's detectors) and 31-speech-acts.md §9. Code: prag/src/silver.rs (span column), prag/data/cues.tsv, world/src/talk.rs. Measures: precision of harvested cues on the silver, act/hedge macro-F1, cross-genre politeness accuracy.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written by Mova (claude-opus-5-5) from the paper text. Original — aclanthology.org/2025.findings-acl.970.pdf

Summary

The paper addresses explainable politeness detection in text. Linear models (Logistic Regression, SVM) with interpretable features perform well on data from the domain they were trained on. They transfer poorly to other domains, however, because they rely on domain-specific vocabulary. The author proposes building features on speech acts instead of individual words. GPT-4 identifies fragments in requests (words, phrases, sentences) that belong to 10 categories: gratitude, greeting, apology, hedging, request, and others. For each fragment, a contextual vector is computed as the mean of its BERT embeddings. The fragments of each category are then clustered by a decision-tree-based algorithm (Aljanaideh et al., 2020), and the cluster IDs become features for the classifier. The models were trained on the Wikipedia portion of the Stanford Politeness Corpus and tested cross-domain on Stack Exchange. Speech act clusters combined with word features achieved the highest accuracy among explainable models in the cross-domain setting. In-domain, they are outperformed by word clusters. The method groups fragments with the same intent even when they share no words. The work shows that LLMs can improve transparent models.

Key points

  • Data: Stanford Politeness Corpus, 2,176 requests from Wikipedia and 3,302 from Stack Exchange. The train/dev/test split is the same as in Aljanaideh et al. (2020).
  • The author manually checked 100 fragments: GPT-4 identified the speech act correctly in 83% of cases. The most frequent error is labeling a direct question as hedging. Among the extracted fragments, direct questions (39%), hedging (18%) and requests (11%) predominate.
  • The best cross-domain result among explainable models is 71.3%. It was achieved by LR on a combination of speech act clusters, word clusters and unigrams. This is 4.1 pp higher than the strongest baseline (unigrams + Politeness Strategies, SVM, 67.2%).
  • Speech act clusters on their own are slightly weaker than the 21 Politeness Strategies features in both settings. In-domain, the best explainable model is word clusters + Politeness Strategies (SVM, 85.1%).
  • Fine-tuned BERT-base remains the most accurate (89.1% in-domain, 75.5% cross-domain), but its decisions are not explainable. Limitations of the method: the act categories must be defined manually, the data are English only, and the annotators are from the US only.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.970

@inproceedings{aljanaideh2025speech,
  title = {{Speech Act Patterns for Improving Generalizability of Explainable Politeness Detection Models}},
  author = {Aljanaideh, Ahmad},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.970},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.970}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report. OpenAI. find our card
  2. Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M Alonso-Moral, Roberto Con-falonieri, Riccardo Guidotti, Javier Del Ser, Natalia Díaz-Rodríguez, and Francisco Herrera. 2023. Explainable Artificial Intelligence (XAI): What we Know and What is Left to Attain Trustworthy Artificial Intelligence. Information fusion, 99:101805. ↗
  3. Ahmad Aljanaideh, Eric Fosler-Lussier, and Marie-Catherine de Marneffe. 2020. Contextualized Embeddings for Enriching Linguistic Analyses on Politeness. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2181– 2190. ↗ our card
  4. John Langshaw Austin. 1975. How to do Things With Words. Harvard University Press. find
  5. Tom B Brown. 2020. Language Models are Few-shot Learners. Advances in Neural Information Processing Systems. find
  6. Feng Chen, Liqin Wang, Julie Hong, Jiaqi Jiang, and Li Zhou. 2024. Unmasking Bias in Artificial Intelligence: A Systematic Review of Bias Detection and Mitigation Strategies in Electronic Health Record-based Models. Journal of the American Medical Informatics Association, 31(5):1172–1183. ↗
  7. Tobias Clement, Nils Kemmerzell, Mohamed Abdelaal, and Michael Amberg. 2023. XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process. Machine Learning and Knowledge Extraction, 5(1):78–108. ↗
  8. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A Computational Approach to Politeness with Application to Social Factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of The 2019 Conference of the North American Chapter of the Association for Computational Linguistics. ↗
  10. Vikas Hassija, Vinay Chamola, Atmesh Mahapatra, Ab-hinandan Singal, Divyansh Goel, Kaizhu Huang, Simone Scardapane, Indro Spinelli, Mufti Mahmud, and Amir Hussain. 2024. Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence. Cognitive Computation, 16(1):45–74. ↗
  11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced BERT with Disentangled Attention. arXiv preprint arXiv:2006.03654. ↗ our card
  12. Yeo Jin Kim, Halim Acosta, Wookhee Min, Jonathan Rowe, Bradford Mott, Snigdha Chaturvedi, and James Lester. 2024. Dual process masking for dialogue act recognition. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15270–15283. ↗
  13. Ruizhe Li, Chenghua Lin, Matthew Collinson, Xiao Li, and Guanyi Chen. 2018. A dual-attention hierarchical recurrent neural network for dialogue act classification. Conference on Computational Natural Language Learning (CoNLL. ↗
  14. Jionghao Lin, Wei Tan, Lan Du, Wray Buntine, David Lang, Dragan Gašević, and Guanliang Chen. 2023. Enhancing educational dialogue act classification with discourse context and sample informativeness. IEEE Transactions on Learning Technologies. ↗
  15. Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2020. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23(1):18. ↗
  16. Yinhan Liu. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. ↗
  17. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine Learning in Python. the Journal of machine Learning research, 12:2825–2830. find
  18. Alec Radford. 2018. Improving Language Understanding by Generative Pre-training. find
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  20. Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-aware Self-attention. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). ↗ our card
  21. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2019. Deep dialog act recognition using multiple token, segment, and context information representations. Journal of Artificial Intelligence Research, 66:861–899. ↗
  22. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card