Elamur Research⌕
☰
home / papers / 2020 / A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations

A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations

Paper · 2020 · Motoki Taniguchi, Yoshihiro Ueda, Tomoki Taniguchi et al.

Summary

Recognizing dialogue acts (DAs) in asynchronous conversations such as e-mail is hard. Topics interleave, reply structure is often missing, and existing annotated corpora are small, tied to one domain, or labelled at a single granularity. The authors built a corpus from the Enron e-mail dataset: they rebuilt threads from forward/reply markers and annotated more than 35,000 sentences in over 6,000 messages from more than 2,000 threads. Every sentence and every message gets one main communicative function, taken from a trimmed version of the ISO 24617-2 standard. As baselines they tested LSTM/CRF-style neural models from earlier work and BERT. BERT came out on top among the models but stayed below expert annotators, so the task is far from solved. Training sentence-level and message-level prediction jointly helped a lot when labelled data was scarce. A model trained on this corpus also transferred well to other asynchronous datasets (BC3 e-mail, QC3 forum). This suggests the domain-neutral tags give a reusable resource for analysing conversations.

Key points

  • Data source and thread reconstruction: the corpus comes from the Enron e-mail dataset, which has no thread structure.
    • Threads were rebuilt by splitting messages at "original message" and "forwarded by" templates.
    • Duplicate threads were removed, as were threads with only one message or more than 20.
  • Tag set: a restricted ISO 24617-2 communicative-function taxonomy, reduced based on tag frequencies in a small trial annotation:
    • Request, Suggestion, Commissive, Question, Answer and Inform;
    • plus None (greetings, jokes) and Others (lists, tables, signatures) to cope with noisy e-mail text.
  • Annotation and human ceiling: crowd workers produced the labels, and three experts annotated data for a human-performance comparison.
    • Experts clearly beat BERT.
    • BERT scored about the same on crowd-labelled and expert-labelled data, which supports using crowd data for evaluation.
  • Domain transfer (BC3, QC3): zero-shot transfer from this corpus beat in-domain training (SELF) by 8.7 F1 on BC3 and 11.2 F1 on QC3.
    • Fine-tuning on the target domain added another 12.1 and 10.9 points.
    • When the source corpus was shrunk to the target corpus size, zero-shot fell below SELF.
  • Multi-task learning: with 500 training messages, one BERT model predicting both levels (MTL-FULL) beat separate models by 24.9 F1 at sentence level and 16.6 F1 at message level.
    • The advantage shrank toward zero as training data grew.

Interesting ideas

  • I1. Two-granularity labels: labelling the same text at two nested levels (sentence and message) with the same tag inventory gives a free auxiliary task, which is especially valuable when labels are scarce.
  • I2. A standard, application-neutral scheme: using a general standard (ISO 24617-2), pruned by observed frequency in a pilot run, keeps the label set small yet portable instead of inventing domain-specific tags.
  • I3. Explicit junk labels: adding None and Others for greetings, signatures and lists stops noisy segments from being forced into real categories.
  • I4. Size-matched control: to explain a surprising transfer gain, the authors shrank the source corpus to the target's size and showed that sheer data volume drove the effect.
  • I5. Checking crowd labels against an expert subset: comparing model scores on crowd labels and on a small expert-labelled subset tests whether cheap labels are good enough for evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act detection in prag could tag both clauses and whole utterances or paragraphs with one inventory and train or check them jointly, so the paragraph act constrains clause acts and vice versa.
  • I2 (frames): Mova's speech-act layer could anchor on the ISO 24617-2 communicative functions, cut to what actually appears in our texts, so results stay comparable with outside corpora.
  • I3 (tokens): segmentation could mark signatures, tables and boilerplate as a separate class before tagging, so they never reach the parser as fake sentences.
  • I4 (learn): when silver data from one domain seems to help another, rerun with a size-matched subset as a negative control to separate data quantity from real domain fit.
  • I5 (learn): silver data agreed by witness parsers could be validated the same way, by checking that model rankings hold on a small hand-checked gold subset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tokyo Metropolitan UniversityTokyo · Japan5 works · 333 citations🏢Fuji XeroxTokyo · Japan2 works · 398 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Motoki Taniguchi, Yoshihiro Ueda, Tomoki Taniguchi and Tomoko Ohkuma. 2020. A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.436

@inproceedings{taniguchi2020large,
  title = {{A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations}},
  author = {Motoki Taniguchi, Yoshihiro Ueda, Tomoki Taniguchi and Tomoko Ohkuma},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.436},
  url = {https://doi.org/10.18653/v1/2020.coling-main.436}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2012. Classifying user messages for managing web forum data. In WebDB. find
  2. Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions – can dialog acts of individual messages help? In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2127–2131, Doha, Qatar, October. Association for Computational Linguistics.
  3. Harry Bunt, Jan Alexandersson, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Volha Petukhova, Andrei Popescu-Belis, and David Traum. 2012. ISO 24617-2: A semantically-based standard for dialogue annotation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 430–437, Istanbul, Turkey, May. European Language Resources Association (ELRA).
  4. Vitor R. Carvalho and William W. Cohen. 2005. On the collective classification of email “speech acts”. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’05, page 345–352, New York, NY, USA. Association for Computing Machinery. find
  5. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to classify email into “speech acts”. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 309–316, Barcelona, Spain, July. Association for Computational Linguistics.
  6. Mark G Core and James Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, volume 56, pages 28–35. Boston, MA. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  8. Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Elizabeth Shriberg. 2004. Meeting recorder project: Dialog act labeling guide. Technical report, INTERNATIONAL COMPUTER SCIENCE INST BERKELEY CA. ↗
  9. Donghui Feng, Erin Shaw, Jihie Kim, and Eduard Hovy. 2006. Learning to detect conversation focus of threaded discussions. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 208–215, New York City, USA, June. Association for Computational Linguistics. find
  10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  11. J. J. Godfrey, E. C. Holliman, and J. McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In Proceedings ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520 vol.1, March. find
  12. Liangjie Hong and Brian D. Davison. 2009. A classification-based approach to question answering in discussion boards. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 171–178, New York, NY, USA. Association for Computing Machinery. find
  13. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1250–1259, Singapore, August. Association for Computational Linguistics. find
  14. Shafiq Joty and Enamul Hoque. 2016. Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1746–1756, Berlin, Germany, August. Association for Computational Linguistics.
  15. Shafiq Joty and Tasnim Mohiuddin. 2018. Modeling Speech Acts in Asynchronous Conversations: A Neural-CRF Approach. Computational Linguistics, 44(4):859–894, December. ↗
  16. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2013. Topic segmentation and labeling in asynchronous conversations. J. Artif. Int. Res., 47(1):521–573, May. ↗
  17. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual. Technical report, UC Boulder & SRI International. find
  18. Su Nam Kim, Li Wang, and Timothy Baldwin. 2010. Tagging and linking web forum posts. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 192–202, Uppsala, Sweden, July. Association for Computational Linguistics. find
  19. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning, pages 217–226. Springer. find
  20. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  21. Annie Louis and Shay B. Cohen. 2015. Conversation trees: A grammar model for topic structure in forums. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1543–1553, Lisbon, Portugal, September. Association for Computational Linguistics. find
  22. Tasnim Mohiuddin, Thanh-Tung Nguyen, and Shafiq Joty. 2019. Adaptation of hierarchical structured models for speech act recognition in asynchronous conversation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1326–1336, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  23. Tatsuro Oya and Giuseppe Carenini. 2014. Extractive summarization and dialogue act modeling on email threads: An integrated probabilistic approach. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 133–140, Philadelphia, PA, U.S.A., June. Association for Computational Linguistics.
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics.
  25. Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal-oriented dialogues (MultiDoGO): Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4526–4536, Hong Kong, China, November. Association for Computational Linguistics. find
  26. Sujith Ravi and Jihie Kim. 2007. Profiling student interactions in threaded discussions with speech act classifiers. In Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, page 357–364, NLD. IOS Press. find
  27. Emanuel A. Schegloff and Harvey Sacks. 1973. Opening up closings. Semiotica, 8(4):289 – 327. ↗
  28. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97–100, Cambridge, Massachusetts, USA, April 30 - May 1. Association for Computational Linguistics. ↗
  29. Jan Ulrich, Gabriel Murray, and Giuseppe Carenini. 2008. A publicly available annotated corpus for supervised email summarization. In Proc. of AAAI email-2008 workshop, chicago, usa. find
  30. Amy X. Zhang, Bryan Culbertson, and Praveen Paritosh. 2017. Characterizing online discussion using coarse discourse sequences. In Proceedings of the 11th International AAAI Conference on Weblogs and Social Media, ICWSM ’17. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taniguchi-2020-large-scale-corpus-e-mail/