Elamur Research⌕
☰
home / papers / 2018 / Frustrated, Polite, or Formal: Quantifying Feelings and Tone in Email

Frustrated, Polite, or Formal: Quantifying Feelings and Tone in Email

Paper · 2018 · Niyati Chhaya, Kushal Chawla, Tanya Goyal et al.

Summary

The paper asks whether frustration, a fine-grained affect common in workplace exchanges, can be measured from email text alone. Earlier frustration work relied on speech, video or physiological sensors. The authors took a subset of the ENRON corpus and used Amazon Mechanical Turk to collect crowd ratings for about 1,050 emails on three tone dimensions: frustration, formality and politeness. Each email's averaged score became a continuous gold label. They then trained regression and classification models on four groups of features: lexical, syntactic, "derived", and word-level affect features drawn from lexicons such as PERMA, ANEW and Emolex. These were compared with a model built on ENRON-trained word embeddings and with prior formality and politeness systems. Ridge regression over the full hand-crafted feature set did best, with an MSE of about 0.018 against human frustration scores. The embedding model lagged behind. The results show that affect lexicon features carry much of the signal for frustration. The paper is presented as the first computational model of frustration in written text.

Key points

  • Data: 960 emails in the main study plus 90 in a pilot, drawn from ~5,000 ENRON emails. Emails average 4.22 sentences and 77.5 words. 69 Turkers rated them, about 10 ratings per email, spending roughly 28 seconds on each.
  • Rating scales: Frustration used a 3-point scale (0 = not frustrated, −1, −2). Formality and politeness used 5-point Likert scales from −2 to +2. Definitions were left to annotators' own perception.
  • Regression: Using affect, syntactic and lexical features, the Ridge model beat Linear, Lasso and SVR. On the email data it was comparable to the Pavlick & Tetreault (formality) and Danescu-Niculescu-Mizil et al. (politeness) baselines.
  • Classification: As a binary task with the positive class oversampled, Random Forest reached accuracy of about 0.85–0.86 (precision 0.88, recall 0.85, F1 0.85).
  • Correlations and features: The three tone dimensions correlate weakly and negatively with each other (Pearson roughly −0.13 to −0.25). F-regression tests mark PERMA negative-affect features and Emolex sadness as highly significant for frustration.

Interesting ideas

  • I1. Treat a subjective tone as whatever the crowd perceives, and use the mean of several untrained raters as a graded target instead of forcing a strict definition; this sidesteps theoretical disputes and still yields learnable labels.
  • I2. Small, interpretable word-level affect lexicons (PERMA, ANEW, Emolex) beat dense embeddings on a modest dataset, so explicit lexical resources can outperform opaque representations when data is scarce.
  • I3. Model several tone axes side by side and check how they correlate, which reveals that tone is multidimensional rather than one polarity scale.
  • I4. Use per-feature F-regression significance tests to explain which features drive each dimension, rather than reporting only aggregate error.
  • I5. Break tone scores down by topic (LDA + KMeans clusters) to show that baseline frustration differs by domain; sports talk behaved unlike the other topics.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Disagreeing LLM teacher tiers or witness parsers could be averaged into soft confidence scores for pragmatic labels such as speech-act force, rather than collapsed into one hard silver tag.
  • I2 (world): Valence/arousal/dominance and emotion values could be compiled into the lexical knowledge layer, so every tone judgement traces back to specific words.
  • I3 (frames): The prag crate could tag utterances with separate frustration, formality and politeness scores, which would help irony and implicature detection when, for example, a formal surface hides a frustrated message.
  • I4 (learn): The same per-feature significance reporting could be added to evaluation so each learned repair rule or feature is justified by a measurable contribution.
  • I5 (answer): Readers answering questions about a character's feelings could calibrate emotional cues against the genre or topic of the text, avoiding misreadings of normal domain vocabulary as distress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations🎓Indian Institute of Technology KharagpurKharagpur · India10 works · 634 citations

Also on the byline: Indian Institute of Technology Guwahati.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Social Psychology · 129 works

Cite

Cite as Niyati Chhaya, Kushal Chawla, Tanya Goyal, Projjal Chanda and Jaya Singh. 2018. Frustrated, Polite, or Formal: Quantifying Feelings and Tone in Email. Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-1111

@inproceedings{chhaya2018frustrated,
  title = {{Frustrated, Polite, or Formal: Quantifying Feelings and Tone in Email}},
  author = {Niyati Chhaya, Kushal Chawla, Tanya Goyal, Projjal Chanda and Jaya Singh},
  booktitle = {Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media},
  year = {2018},
  doi = {10.18653/v1/w18-1111},
  url = {https://doi.org/10.18653/v1/w18-1111}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078. ↗ our card
  2. Nesreen Kamel Ahmed and Ryan Anthony Rossi. 2015. Interactive visual graph analytics on the web. In ICWSM, pages 566–569. find
  3. Sufal Das and Hemanta Kumar Kalita. 2017. Sentiment analysis for web-based big data: A survey. International Journal, 8(5). ↗
  4. I Elaine Allen and Christopher A Seaman. 2007. Likert scales and data analyses. Quality progress, 40(7):64. find
  5. Jana Diesner and Craig S Evans. 2015. Little bad concerns: Using sentiment analysis to assess structural balance in communication networks. In Advances in Social Networks Analysis and Mining (ASONAM), 2015 IEEE/ACM International Conference on, pages 342–348. IEEE. find
  6. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic acquisition of lexical formality. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 90–98. Association for Computational Linguistics. find
  7. Jana Diesner, Terrill L Frantz, and Kathleen M Carley. 2005. Communication networks from the enron email corpus it’s always about the people. enron is no different. Computational & Mathematical Organization Theory, 11(3):201–228. ↗
  8. Judee K. Burgoon and Jerold L. Hale. 1984. The fundamental topoi of relational communication. Communication Monographs, 51(3):193–214. ↗
  9. Sidney K. D’Mello, Scotty D. Craig, Amy Witherspoon, Bethany McDaniel, and Arthur Graesser. 2008. Automatic detection of learner’s affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1):45–80. ↗
  10. Rafael A Calvo and Sidney D’Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on affective computing, 1(1):18–37. ↗
  11. Paul Ekman. 1992. An argument for basic emotions. Cognition & Emotion, 6(3-4):169–200. ↗ our card
  12. Anurat Chapanond, Mukkai S Krishnamoorthy, and Bülent Yener. 2005. Graph theoretic and spectral analysis of enron email data. Computational & Mathematical Organization Theory, 11(3):265–281. ↗
  13. Sanaz Jabbari, Ben Allison, David Guthrie, and Louise Guthrie. 2006. Towards the orwellian nightmare: separation of business and personal emails. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 407–411. Association for Computational Linguistics. find
  14. Leon Ciechanowski, Aleksandra Przegalinska, and Krzysztof Wegner. 2018. The Necessity of New Paradigms in Measuring Human-Chatbot Interaction. Springer International Publishing, Cham. find
  15. Rohit J Kate, Xiaoqiang Luo, Siddharth Patwardhan, Martin Franz, Radu Florian, Raymond J Mooney, Salim Roukos, and Chris Welty. 2010. Learning to predict readability using diverse linguistic features. find
  16. William W Cohen. 2009. Enron email dataset. find
  17. Cristina Conati and Heather Maclaren. 2009. Empirically building and evaluating a probabilistic model Brandon Oselio, Alex Kulesza, and Alfred O Hero. 2014. Multi-layer graph analysis for dynamic social networks. IEEE Journal of Selected Topics in Signal Processing, 8(4):514–523. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 546–554. Association for Computational Linguistics. find
  18. Jasleen Kaur and Jatinderkumar R Saini. 2014. Emotion detection and sentiment analysis in text corpus: a differential study with informal and formal writing styles. International Journal of Computer Applications, 101(9). ↗
  19. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61–74. ↗ our card
  20. Kelly Peterson, Matt Hohensee, and Fei Xia. 2011. Email formality in the workplace: A case study on the enron corpus. In Proceedings of the Workshop on Languages in Social Media, pages 86–95. Association for Computational Linguistics. find
  21. Shibamouli Lahiri. 2015. SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature. CoRR, abs/1506.02306. ↗ our card
  22. Sisi Liu and Ickjai Lee. 2015. A hybrid sentiment analysis framework for large email data. In Intelligent Systems and Knowledge Engineering (ISKE), 2015 10th International Conference on, pages 324–330. IEEE. find
  23. Vinodkumar Prabhakaran, Huzaifa Neralwala, Owen Rambow, and Mona T Diab. 2012. Annotations for power relations on email threads. In LREC, pages 806–811. find
  24. Scott W. McQuiggan, Sunyoung Lee, and James C. Lester. 2007. Early Prediction of Student Frustration. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  25. Jennifer Sabourin, Bradford Mott, and James C. Lester. 2011. Modeling Learner Affect with Theoretically Grounded Dynamic Bayesian Networks. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  26. Albert Mehrabian. 1980. Basic dimensions for a general psychological theory implications for personality, social, environmental, and developmental studies. find
  27. H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9):e73791. ↗
  28. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗ our card
  29. Martin EP Seligman. 2011. Flourish: a visionary new understanding of happiness and well-being. Policy, 27(3):60–1. find
  30. Christopher A Miller and Jeffrey Rye. 2012. Power and politeness in interactions: Admire-a tool for deriving the former from the latter. In Social Informatics (SocialInformatics), 2012 International Conference on, pages 177–184. IEEE. find
  31. Martin EP Seligman. 2012. Flourish: A visionary new understanding of happiness and well-being. Simon and Schuster. find
  32. R Miller and EYA Charles. 2016. A psychological based analysis of marketing email subject lines. In Advances in ICT for Emerging Regions (ICTer), 2016 Sixteenth International Conference on, pages 58–65. IEEE. find
  33. Jitesh Shetty and Jafar Adibi. 2005. Discovering important nodes through graph entropy the case of enron email database. In Proceedings of the 3rd international workshop on Link discovery, pages 74–81. ACM. find
  34. Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. ↗ our card
  35. Robert J Sigley. 1997. Text categories and where you can stick them: a crude formality index. International Journal of Corpus Linguistics, 2(2):199–237. ↗
  36. Lisa M Vizer, Lina Zhou, and Andrew Sears. 2009. Automated stress detection using keystroke and linguistic features: An exploratory study. International Journal of Human-Computer Studies, 67(10):870– 886. ↗
  37. Saif M Mohammad and Tony Wenda Yang. 2011. Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis, pages 70–79. Association for Computational Linguistics. find
  38. Hua Wang, Helmut Prendinger, and Takeo Igarashi. 2004. Communicating emotions in online chat using physiological sensors and animated text. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’04, pages 1171–1174, New York, NY, USA. ACM. find
  39. Myriam D Munezero, Calkin Suero Montero, Erkki Sutinen, and John Pajunen. 2014. Are they different? affect, feeling, emotion, sentiment, and opinion detection in text. IEEE transactions on affective computing, 5(2):101–111. ↗
  40. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior Research Methods, 45(4):1191–1207. ↗
  41. Yingjie Zhou, Kenneth R Fleischmann, and William A Wallace. 2010. Automatic text analysis of values in the enron email dataset: Clustering a social network using the value patterns of actors. In System Sciences (HICSS), 2010 43rd Hawaii International Conference on, pages 1–10. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chhaya-2018-frustrated-polite-or-formal-quantifying/