Elamur Research⌕
☰
home / papers / 2022 / TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages

TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages

Summary

Computational work on politeness has focused almost entirely on English, which leaves open whether politeness cues carry over to other languages and cultures. Srinivasan and Choi tackle this question by building TyDiP, a corpus of requests originally composed in each language on Wikipedia User talk pages, covering nine languages (in alphabetical order, Afrikaans, French, Hindi, Hungarian, Korean, Russian, Spanish, Tamil and Vietnamese). The languages span five scripts and eight families. Bilingual crowdworkers on Prolific rated each request on a graded scale. Workers were screened with a translation test and a qualification step that checked their labels against confident predictions from a multilingual classifier.

The authors then test zero-shot transfer. XLMR fine-tuned only on the English politeness data of Danescu-Niculescu-Mizil et al. (2013) performs competitively in every language, and a large prompted language model also does well. Translating inputs into English first lowers accuracy. Even so, the best models remain well below estimated human accuracy. Two further analyses project English politeness strategies into the nine languages and compare politeness with formality. Together, the dataset and lexicons support multilingual evaluation and the study of cross-cultural pragmatics.

Key points

  • TyDiP has 500 requests per language (4.5K total), half of them reserved for evaluation. Each request has three annotations on a −3 to 3 scale in 0.5 steps.
  • Raters agreed more on their native-language requests than on English ones, with mean pairwise Spearman correlations of 0.46 versus 0.36.
  • The gap between the best model and estimated human accuracy exceeds 10 points in six of the nine languages. Translate-to-English pipelines using the Google Translate API underperform direct zero-shot classification.
  • The English politeness strategy lexicon was mapped into all nine languages using machine translation and word alignment (Dou and Neubig, 2021). Most strategies keep a consistent correlation with politeness across languages, with a few exceptions.
  • Politeness and formality turn out to be distinct notions. About 80% of English informal/formal pairs get the same politeness label, yet in roughly 70% of pairs the formal version receives a higher politeness probability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.420 · arXiv 2211.16496

@inproceedings{srinivasan2022tydip,
  title = {{TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages}},
  author = {Srinivasan, Anirudh and Choi, Eunsol},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.420},
  eprint = {2211.16496},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.420}
}

References

40 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Malika Aubakirova and Mohit Bansal. 2016. Interpreting neural networks to improve politeness comprehension. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2035–2041, Austin, Texas. Association for Computational Linguistics. find our card
  2. Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021. Olá, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages… find
  3. Penelope Brown and Stephen C Levinson. 1978. Universals in language usage: Politeness phenomena. In Questions and politeness: Strategies in social interaction, pages 56–311. Cambridge University Press. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  5. Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wa-hab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Auguste Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios Gonzales, Isabel Papadimitriou, S… find our card
  6. J. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. find our card
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… find
  9. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… find our card
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. find
  12. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia.… find
  13. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. A massive collection of cross-lingual web-document pairs. In EMNLP. find
  14. Liye Fu, Susan Fussell, and Cristian Danescu-Niculescu-Mizil. 2020. Facilitating the communication of politeness through fine-grained paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5127–5140, Online. Association for Computationa… find our card
  15. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… find our card
  16. Nada Korac-Kakabadse, Alexander Kouzmin, Andrew Korac-Kakabadse, and Lawson Savery. 2001. Low-and high-context communication patterns: towards mapping cross-cultural encounters. Cross cultural management: An international journal. find
  17. Shibamouli Lahiri. 2016. Squinky! a corpus of sentence-level formality, informativeness, and implicature. find
  18. Robin Lakoff. 1977. What you can do with words: Politeness, pragmatics and performatives. In Proceedings of the Texas conference on performatives, presuppositions and implicatures, pages 79–106. ERIC. find
  19. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. Computational Linguistics, 38:799–825. find
  20. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… find our card
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  22. S. Longpre, Yi Lu, and Joachim Daiber. 2021. Mkqa: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389– 1406. find
  23. Sara Mills. 2003. Gender and politeness. find
  24. Shuyo Nakatani. 2010. Language detection library for java. find
  25. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find our card
  26. Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373– 389. find
  27. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. ArXiv, abs/2003.11080. find our card
  28. Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014. The language demographics of Amazon Mechanical Turk. Transactions of the Association for Computational Linguistics, 2:79–92. find
  29. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  30. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61–74. find our card
  31. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35–40, San Diego… find our card
  33. Diogo Silva, David Semedo, and João Magalhães. 2022. Polite task-oriented dialog agents: To generate or to rewrite? In WASSA. find
  34. Rion Snow, Brendan T. O’Connor, Dan Jurafsky, and A. Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP. find
  35. Pedro Ortiz Suarez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. find
  36. Zhixing Tan, Xiangwen Zhang, Shuo Wang, and Yang Liu. 2022. MSP: Multi-stage prompting for making pre-trained language models better translators. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6131–6142, Dublin, Ireland. Asso… find
  37. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15, Punta Cana, Dominican Republic. Association for Computat… find
  38. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In EMNLP. find our card
  39. Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547–8555, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. hi ko es ta f… find
  40. (0.48) (0.48) (0.48) (0.48) (0.46) (0.48) (0.52) (0.48) (0.5) scores. The minimum and maximum scores seem to vary a bit across languages. Some languages like Spanish have a higher median score and a higher number of sentences with a positive scores. (0.44) (0.46) (0.42) (0.39) (0.35) (0.39) (0.4) (0… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/srinivasan-2022-tydip-dataset-politeness-classification-nine/