Elamur Research⌕
☰
home / papers / 2015 / Dialog Act Annotation for Twitter Conversations

Dialog Act Annotation for Twitter Conversations

Summary

Most NLP work on Twitter treats tweets as standalone texts, even though a large share of posts belong to reply threads. The authors examine how German Twitter conversations work by labelling them with dialog acts (DAs), which capture what each utterance does in its conversational context.

They collected threads about the Energiewende (energy turnaround) from 2013 data, covering both long and short discussions. Three minimally trained linguistics students annotated each conversation in WebAnno. They split tweets into segments and labelled each segment using a 57-act schema adapted from DIT++.

  • Agreement: It was moderate on the full schema and noticeably higher once labels were collapsed into ten broad classes. The broad-class level is comparable to earlier studies.
  • Gold standard: The annotations were merged into one, using coarser categories where annotators disagreed.

The analysis shows that a single tweet often contains several consecutive DAs. Labels such as INFORM also cover functions that are quite different from one another. This suggests that Twitter DA schemes need multi-dimensional, segment-level annotation rather than one label per tweet.

Key points

  • Corpus: 1566 tweets in 172 threads. After removing non-German tweets (langid plus CLD2, with manual checks) and their replies, 1213 German tweets remained.
  • Threads: All long threads (over 20 tweets, depth ≥ 5) were included, plus a random sample of short threads (4–5 tweets).
  • Annotation setup: 36 students from a University of Potsdam seminar annotated in WebAnno, with three annotators per conversation. The schema was a DIT++-based taxonomy of 57 DAs.
  • Agreement: multi-π = 0.56 for the full schema and 0.76 for 10 coarse DAs. For comparison, Stolcke et al. reported κ = 0.8 on spoken data with 42 categories.
  • Findings: Tweets often combine several DAs. Annotators struggled to choose between dimensions (e.g., OPEN vs. TOPIC INTRODUCTION). INFORM blends facts, opinions, meta-comments and sarcasm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-4614

@inproceedings{zarisheva2015dialog,
  title = {{Dialog Act Annotation for Twitter Conversations}},
  author = {Zarisheva, Elina and Scheffler, Tatjana},
  booktitle = {Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2015},
  doi = {10.18653/v1/w15-4614},
  url = {https://doi.org/10.18653/v1/w15-4614}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555–596. ↗
  2. John Langshaw Austin. 1975. How to do things with words, volume 367. Oxford university press. find
  3. Harry Bunt, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, et al. 2010. Towards an ISO standard for dialogue act annotation. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). find our card
  4. Jean Carletta, Stephen Isard, Gwyneth Doherty-Sneddon, Amy Isard, Jacqueline C Kowtko, and Anne H Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational linguistics, 23(1):13–31. ↗
  5. Jean Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational linguistics, 22(2):249–254. ↗ our card
  6. Mark Core and James Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, pages 28–35. Boston, MA. find
  7. Eric N. Forsyth and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog. pages 19–26. find
  8. Chris Fournier and Diana Inkpen. 2012. Segmentation similarity and agreement. pages 152–161. find
  9. Jeroen Geertzen, Volha Petukhova, and Harry Bunt. 2008. Evaluating Dialogue Act Tagging with Naive and Expert Annotators. In Proceedings of LREC, pages 1076–1082. find
  10. Courtenay Honey and Susan C Herring. 2009. Beyond microblogging: Conversation and collaboration via twitter. In System Sciences, 2009. HICSS’09. 42nd Hawaii International Conference on, pages 1–10. IEEE. find
  11. Susanne Jekat, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and J. Joachim Quantz. 1995. Dialogue acts in Verbmobil. Technical report, Saarländische Universitäts- und Landesbibliothek. ↗
  12. Marco Lui and Timothy Baldwin. 2012. langid.py: An Off-the-shelf Language Identification Tool. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, (July):25–30. find our card
  13. Iain McCowan, Jean Carletta, W Kraaij, S Ashby, S Bourban, M Flynn, M Guillemot, T Hain, J Kadlec, V Karaiskos, et al. 2005. The AMI meeting corpus. In Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research. find
  14. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In Proceedings of NAACL. find
  15. Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus annotation through crowdsourcing: Towards best practice guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, number 2010, pages 859–866. find
  16. Tatjana Scheffler. 2014. A German Twitter snapshot. In N. Calzolari et al., editor, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language Resources Association (ELRA). find
  17. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
  18. David R Traum. 2000. 20 questions on dialogue act taxonomies. Journal of semantics, 17(1):7–30. ↗
  19. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (System Demonstrations) (ACL 2013), pages 1–6, Stroudsburg, PA, USA, August. Association for Computational Linguistics. find
  20. Renxian Zhang, Dehong Gao, and Wenjie Li. 2011. What are tweeters doing: Recognizing speech acts in twitter. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. (a) Adapted DIT++ taxonomy (1). (b) Adapted DIT++ taxonomy (2). Figure 3: Adapted DIT++ taxonomy. DSM O PEN DSM T OPIC I NTRODUCTION DSM T OPIC S HIFT IT IT IP IT IP I NFORM IT IP I NF A GREEMENT IT IP I NF A NSWER IT IP I NF D ISAGREEMENT IT IP I NF D IS C ORRECTION 1 2 3 4 5 6 7 8 9 Table 8: Annotation confusion matrix (1): Number of segments judged as both indicated dialog act labels by different annotators. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zarisheva-2015-dialog-act-annotation-twitter-conversations/