Elamur Research⌕
☰
home / papers / 2014 / Extractive Summarization and Dialogue Act Modeling on Email Threads: An Integrated Probabilistic Approach

Extractive Summarization and Dialogue Act Modeling on Email Threads: An Integrated Probabilistic Approach

Summary

The paper tests whether recognizing dialogue acts in email threads helps identify the sentences that belong in an extractive summary, a link that earlier work largely ignored. The authors build a supervised sequential model, a DCRF, that predicts summary membership and dialogue act labels together rather than in separate stages. To capture thread structure, they convert each conversation into a fragment quotation graph, following Carenini et al. Each path through the graph becomes one training sequence. Sentences that appear on several paths receive their final label by majority vote. Features include sentence length, relative position, overlap with the thread title and subject, question marks, CC, participation dominance and a simplified ClueWordScore. For dialogue acts, the model also uses abstracted word and POS n-grams. On the BC3 corpus, the joint model clearly beats separate CRF and SVM baselines at selecting summary sentences, and its dialogue act accuracy stays at the same level. This suggests that modeling conversational function jointly is a practical way to improve email summarization.

Key points

  • Data and evaluation. Experiments use the BC3 email corpus (about 1300 sentences). Results are averaged over five runs of 10-fold cross-validation, and significance is checked with a paired t-test.
  • Summarization results. The DCRF reaches an F-measure of 0.485, against 0.428 for the CRF and 0.397 for the SVM. The gain comes mainly from higher recall (0.457).
  • Combined task. Here a sentence counts as correct only if both its summary label and its dialogue act are right. On this measure the DCRF scores F' = 0.352, against 0.324 for the CRF and 0.292 for the SVM, and the differences are significant.
  • Dialogue act results. Micro accuracy is nearly identical across models (DCRF 0.785, CRF 0.779, SVM 0.775). The two sequential models clearly beat the SVM on macro accuracy, but the DCRF's advantage over the CRF is not significant.
  • Rare acts. The R, Su and M acts have only 68, 73 and 97 examples, and all models classify them poorly. The authors attribute this to n-gram sparsity. Q is easy to detect because its features, such as "?", are distinctive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-4318

@inproceedings{oya2014extractive,
  title = {{Extractive Summarization and Dialogue Act Modeling on Email Threads: An Integrated Probabilistic Approach}},
  author = {Oya, Tatsuro and Carenini, Giuseppe},
  booktitle = {Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)},
  year = {2014},
  doi = {10.3115/v1/w14-4318},
  url = {https://doi.org/10.3115/v1/w14-4318}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, and W. Taysom. Plow: A collaborative task learning agent. In AAAI-07, pages 22–26, 2007. find
  2. Giuseppe Carenini, Gabriel Murray, and Raymond Ng. 2011. Methods for Mining and Summarizing Text Conversations. Morgan Claypool. ↗
  3. Giuseppe Carenini, Raymond Ng, and Xiaodong Zhou. 2005. Scalable discovery of hidden emails from large folders. In ACM SIGKDD’05, pages 544–549. find
  4. Giuseppe Carenini, Raymond Ng, and Xiaodong Zhou. 2008. Summarizing Emails with Conversational Cohesion and Subjectivity In proceeding 46th Annual Meetint Assoc.for Computational Linguistics, page 353-361. find
  5. Giuseppe Carenini, Raymond Ng, and Xiaodong Zhou. 2007. Summarizing email conversations with clue words. 16th International World Wide Web Conference (ACM WWW’07). find
  6. Vitor R. Carvalho and William W. Cohen. 2006. Improving ”email speech acts” analysis via n-gram selection. In Proceedings of the HLT-NAACL 2006 Workshop on Analyzing Conversations in Text and Speech, ACTS ’09, pages 35–41, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  7. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to classify email into “speech acts”. In Proceedings of Empirical Methods in Natural Language Processing, pages 309– 316, Barcelona, Spain, July. find our card
  8. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 363-370. find
  9. George Forman. 2003. An extensive empirical study of feature selection metrics for text classification. The Journal of Machine Learning Research, 3:1289–1305. find
  10. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. find
  11. Thorsten Joachims. 1999 Making large-Scale SVM Learning Practical. Advances in Kernel Methods - Support Vector Learning, B. Schölkopf and C. Burges and A. Smola (ed.), MIT-Press, 1999. ↗
  12. Shafiq Joty, Giuseppe Carenini, and Lin, Chin-Yew Lin. 2011. Unsupervised Modeling of Dialog Acts in Asynchronous Conversations. In Proceedings of the twenty second International Joint Conference on Artificial Intelligence (IJCAI) 2011. Barcelona, Spain. find
  13. Shafiq Joty, Giuseppe Carenini, Gabriel Murray, and Raymond Ng. 2009 Finding Topics in Emails: Is LDA enough? NIPS-2009 workshop on applications for topic models: text and beyond. Whistler, Canada. find
  14. McCallum, A. Kachites, 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu. ↗
  15. Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010a. Classifying dialogue acts in 1-to-1 live chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP 2010), pages 862–871, Boston, USA. find
  16. Su Nam Kim, Lawrence Cavedon and Timothy Baldwin (2012) Classifying Dialogue Acts in Multi-party Live Chats, In Proceedings of the 26th Pacific Asia Conference on Language, Information and Computation (PACLIC 26), Bali, Indonesia, pp. 463—472. find our card
  17. Gabriel Murray and Giuseppe Carenini. 2008. Summarizing Spoken and Written Conversations. Empirical Methods in NLP (EMNLP 2008), Waikiki, Hawaii, 2008. find
  18. Gabriel Murray and Giuseppe Carenini. 2010. Summarizing Spoken and Written Conversations. Generating and Validating Abstracts of Meeting Conversations: a User study (INLG 2010), Dublin, Ireland, 2010. find
  19. Gabriel Murray, Renals Steve, and Carletta Jean. 2005a. Extrative summarization of meeting recordings. In Proceeding of Interspeech 2005, Lisbon, Portugal, pages 593-596. find
  20. Owen Rambow, Lokesh Shrestha, John Chen, and Chirsty Lauridsen. 2004. Summarizing email threads. In Proceedings of HLTNAACL 2004. find
  21. Dou Shen, Jian-Tao Sun, Hua Li, Qiang Yang, and Zheng Chen. 2007. Document summarization using conditional random fields. In Proc. of IJCAI, volume 7, 2862–2867. find
  22. Charles Sutton, Khashayar Rohanimanesh, and Andrew McCallum. 2004. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. In Proc. ICML. find
  23. Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini and Raymond Ng. 2013. Dialogue Act Recognition in Synchronous and Asynchronous Conversations. In Proceedings of the SIGDIAL 2013 Conference, pages 117–121, Metz, France. Association for Computational Linguistics. ↗ our card
  24. Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In Proceedings of HLT-NAACL 2003, pp. 252-259. find
  25. Jan Ulrich, Giuseppe Carenini, Gabriel Murray, and Raymond T. Ng: Regression-Based Summarization of Email Conversations. ICWSM 2009 find
  26. Jan Ulrich, Gabriel Murray, and Giuseppe Carenini. 2008. A publicly available annotated corpus for supervised email summarization. AAAI-2008 EMAIL Workshop. find
  27. Martin J. Wainwright, Tommi Jaakkola, and Alan S. Willsky. 2002. Treebased Reparameterization for Approximate Inference on Loopy Graphs. In Advances in Neural Information Processing Systems 14, pages 1001 1008. MIT Press. find

↑ back to top