Elamur Research
☰
home / papers / 2004 / Learning to Classify Email into ``Speech Acts''

Learning to Classify Email into ``Speech Acts''

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W04-3240.pdf

Summary

The paper addresses classifying email by the sender's intent rather than by topic, framed as "email speech acts." The authors build a taxonomy where each act is a verb (e.g., Request, Propose, Amend, Commit, Deliver) paired with a noun describing its object (e.g., information, meeting, task). One message may carry several acts.

They annotated four small, task-oriented corpora: three team subsets of the CSpace corpus, which comes from an MBA management course at Carnegie Mellon, and the PW CALO corpus from a role-play exercise at SRI. Inter-annotator agreement was checked with kappa. Standard text-classification learners were then trained on TFIDF bag-of-words features: SVM, Voted Perceptron, AdaBoost and decision trees. Entity extraction and part-of-speech information gave additional gains.

Several categories proved learnable, with precision above 80% at recall above 50%. Classifiers also carried over reasonably well between disjoint groups of users. The work suggests that email assistants could passively detect requests and commitments, supporting task tracking without forcing users to tag their own messages.

Key points

  • Corpora: three CSpace subsets, N01F3 (341 messages), N02F2 (351) and N03F2 (443), plus PW CALO (222). The full CSpace collection holds about 15,000 emails from 277 students working in roughly 50 teams. CSpace messages average about 1.3 verb acts each.
  • Agreement: kappa on the double-labeled N03F2 ranged from 0.72 (Commit, Propose) to 0.83 (Amend), with Meeting at 0.82. Agreement was higher on messages containing only one act.
  • Excluded verbs: Refuse, Greet and Remind were too rare to train classifiers for. Nested acts such as (commit (deliver information)) were handled only at the top-level verb.
  • Transfer: classifiers trained on one CSpace team and tested on another performed about as well as within-corpus cross-validation. This held for Directive with decision trees, Deliver with Voted Perceptron, and Commit with SVM.
  • Future work: the authors propose using thread context, factored verb–noun models (e.g., mixture models) and segment-level annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

52 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cohen2004learning,
  title = {{Learning to Classify Email into ``Speech Acts''}},
  author = {Cohen, William W. and Carvalho, Vitor R. and Mitchell, Tom M.},
  booktitle = {Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing},
  year = {2004},
  url = {https://aclanthology.org/W04-3240/}
}

References

29 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Argamon, M. Šariç and S. S. Stein. (2003). Style mining of electronic messages for multiple authorship discrimination: first results. Proceedings of the 9th ACM SIGKDD, Washington, D.C. find
  2. V. Bellotti, N. Ducheneaut, M. Howard and I. Smith. (2003). Taking email to task: the design and evaluation of a task management centered email tool. Proceedings of the Conference on Human Factors in Computing Systems, Ft. Lauderdale, Florida. find
  3. D. Blei, T. Griffiths, M. Jordan, and J. Tenenbaum. (2003). Hierarchical topic models and the nested Chinese restaurant process. Advances in Neural Information Processing Systems, 16, MIT Press. find
  4. B. M. Camino, A. E. Millewski, D. R. Millen and T. M. Smith. (1998). Replying to email with structured responses. International Journal of Human-Computer Studies. Vol. 48, Issue 6, pp 763 – 776. find
  5. J. Carletta. (1996). Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, Vol. 22, No. 2, pp 249-254. find our card
  6. V. R. Carvalho & W. W. Cohen (2004). Learning to Extract Signature and Reply Lines from Email. To appear in Proc. of the 2004 Conference on Email and Anti-Spam. Mountain View, California. find
  7. W. W. Cohen. (1996). Learning Rules that Classify E-Mail. Proceedings of the 1996 AAAI Spring Symposium on Machine Learning and Information Access, Palo Alto, California. find
  8. W. W. Cohen. (2004). Minorthird: Methods for Identifying Names and Ontological Relations in Text using Heuristics for Inducing Regularities from Data, http://minorthird.sourceforge.net. ↗
  9. M. Finke, M. Lapata, A. Lavie, L. Levin, L. May-fieldTomokiyo, T. Polzin, K. Ries, A. Waibel and K. Zechner. (1998). CLARITY: Inferring Discourse Structure from Speech. In Applying Machine Learning to Discourse Processing, AAAI'98. find
  10. F. Flores, and J.J. Ludlow. (1980). Doing and Speaking in the Office. In: G. Fick, H. Spraque Jr. (Eds.). Decision Support Systems: Issues and Challenges, Pergamon Press, New York, pp. 95-118. find
  11. Y. Freund and R. Schapire. (1999). Large Margin Classification using the Perceptron Algorithm. Machine Learning 37(3), 277—296. find
  12. T. Joachims. (2001). A Statistical Learning Model of Text Classification with Support Vector Machines. Proc. of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001. R. E. Kraut, S. R. Fussell, F. J. Lerch, and J. A. Espinosa. (under review). Coordination in teams: evidence from a simulated management game. To appear in the Journal of Organizational Behavior. find
  13. D. D. Lewis. (1992). Representation and Learning in Information Retrieval. PhD Thesis, No. 91-93, Computer Science Dept., Univ of Mass at Amherst A. McCallum, D. Freitag and F. Pereira. (2000). Maximum Entropy Markov Models for Information Extraction and Segmentation. Proc. of the 17th Int’l Conf. on Machine Learning, Nashville, TN. find
  14. B. Pang, L. Lee and S. Vaithyanathan. (2002). Thumbs up? Sentiment Classification using Machine Learning Techniques. Proc. of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 79-86. find
  15. A. E. Milewski and T. M. Smith. (1997). An Experimental System For Transactional Messaging. Proc. of the international ACM SIGGROUP conference on Supporting group work: the integration challenge, pp. 325-330. find
  16. H. Murakoshi, A. Shimazu and K. Ochimizu. (1999). Construction of Deliberation Structure in Email Communication,. Pacific Association for Computational Linguistics, pp. 16-28, Waterloo, Canada. find
  17. A. Ratnaparkhi. (1999). Learning to Parse Natural Language with Maximum Entropy Models. Machine Learning, Vol. 34, pp. 151-175. find
  18. J. D. M. Rennie. (2000). Ifile: An Application of Machine Learning to Mail Filtering. Proc. of the KDD-2000 Workshop on Text Mining, Boston, MA. find
  19. M. Sahami, S. Dumais, D. Heckerman and E. Horvitz. (1998). A Bayesian Approach to Filtering Junk E-Mail. AAAI'98 Workshop on Learning for Text Categorization. Madison, WI. find
  20. M. Schoop. (2001). An introduction to the language-action perspective. SIGGROUP Bulletin, Vol. 22, No. 2, pp 3-8. find
  21. S. Scott and S. Matwin. (1999). Feature engineering for text classification. Proc. of 16th International Conference on Machine Learning, Bled, Slovenia. find
  22. J. R. Searle. (1975). A taxonomy of illocutionary acts. In K. Gunderson (Ed.), Language, Mind and Knowledge, pp. 344-369. Minneapolis: University of Minnesota Press. find
  23. R. B. Segal and J. O. Kephart. (2000). Swiftfile: An intelligent assistant for organizing e-mail. In AAAI 2000 Spring Symposium on Adaptive User Interfaces, Stanford, CA. find
  24. Y. Yang. (1999). An Evaluation of Statistical Approaches to Text Categorization. Information Retrieval, Vol. 1, Numbers 1-2, pp 69-90. find
  25. S. Wermter and M. Löchel. (1996). Learning dialog act processing. Proc. of the International Conference on Computational Linguistics, Kopenhagen, Denmark. find
  26. R. E. Schapire and Y. Singer. (1998). Improved boosting algorithms using confidence-rated predictions. The 11th Annual Conference on Computational Learning Theory, Madison, WI. find
  27. H. Wiegend, G. Goldkuhl, and A. de Moor. (2003). Proc. of the Eighth Annual Working Conference on Language-Action Perspective on Communication Modelling (LAP 2003), Tilburg, The Netherlands. find
  28. J. Wiebe, R. Bruce, M. Bell, M. Martin and T. Wilson. (2001). A Corpus Study of Evaluative and Speculative Language. Proceedings of the 2nd ACL SIGdial Workshop on Discourse and Dialogue. Aalborg, Denmark. find
  29. T. Winograd. 1987. A Language/Action Perspective on the Design of Cooperative Work. Human-Computer Interactions, 3:1, pp. 3-30. S. Whittaker, Q. Jones, B. Nardi, M. Creech, L. Terveen, E. Isaacs and J. Hainsworth. (in press). Using Personal Social Networks to Organize Communication in a Social desktop. To appear in Transactions on Human Computer Interaction. find