Elamur Research⌕
☰
home / papers / 2018 / Multi-task dialog act and sentiment recognition on Mastodon

Multi-task dialog act and sentiment recognition on Mastodon

Summary

Twitter corpora quickly become impossible to reproduce: licensing terms restrict long-term storage, and deleted tweets vanish from the servers. The authors propose Mastodon as an alternative. It is a decentralized, Twitter-like network whose posts are usually published under Creative Commons terms, so they can be redistributed. They collected English posts from one Mastodon instance, rebuilt the reply threads into dialogues, and had two linguists annotate every post with a sentiment label and a dialog act label. They then trained a two-level hierarchical recurrent network that predicts both labels jointly. Experiments in which one task had only a small amount of annotation showed that knowledge transfers between the tasks in both directions. The weak task approached the performance it reached with full supervision and clearly beat the setting where both tasks had little data. Further analysis found that the two phenomena are only loosely correlated overall, although some specific patterns are strong. The annotated corpus and the code are released openly, which supports reproducible social-media NLP research.

Key points

  • Data: About 800,000 posts were crawled from octodon.social and filtered to English with langdetect. Reply trees were linearized so that each branch becomes one dialogue. The training set has 239 dialogues (1075 posts), the test set has 266 dialogues (1142 posts), and the vocabulary contains 5330 words.
  • Annotation: Sentiment uses three classes: positive 26%, negative 31% and neutral 43%. Dialog acts were derived from the Switchboard scheme, adapted for social media, and informed by the ISO standard. The initial 27 dialog act tags were merged into 15. The two annotators agreed on 88.6% of dialog act labels and 90.2% of sentiment labels, and Cohen's kappa for dialog acts is 85.1%.
  • Model: A bi-LSTM encodes each post into a vector. A plain RNN then runs over the posts of a dialogue. Two MLP heads, one per task, are trained with separate cross-entropy losses.
  • Transfer: When one task is limited to 38 annotated training dialogues, joint training with the fully labeled task recovers most of the lost performance. This holds in both directions.
  • Analysis: Sentiment shifts less often than dialog acts within a dialogue. Agreements tend to keep the previous sentiment, while disagreements tend to change it. Negative sentiment appears more often early in dialogues and positive sentiment toward the end. The corpus and code are available at github.com/cerisara/DialogSentimentMastodon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

21 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 20 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1807.05013

@inproceedings{cerisara2018multi,
  title = {{Multi-task dialog act and sentiment recognition on Mastodon}},
  author = {Cerisara, Christophe and Jafaritazehjani, Somayeh and Oluokun, Adedayo and Le, Hoa T.},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  eprint = {1807.05013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.05013}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors. 2017. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, August. find
  2. Kristy Boyer, Joseph Grafsgaard, Eun Young Ha, Robert Phillips, and James Lester. 2011. An affect-enriched dialogue act classification model for task-oriented dialogue. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1190–1199, Portland, Oregon, USA, June. Association for Computational Linguistics. find our card
  3. Harry Bunt, Volha Petukhova, David Traum, and Jan Alexandersson. 2017. Dialogue act annotation with the iso 24617-2 standard. In Multimodal Interaction with W3C Standards, pages 109–135. Springer. ↗
  4. Chlo Clavel and Zoraida Callejas. 2016. Sentiment analysis: from opinion mining to human-agent interaction. IEEE Transactions on Affective Computing, pages 74–93. ↗
  5. Mark G Core and James Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, volume 56. Boston, MA. find
  6. Eric N Forsythand and Craig H Martell. 2007. Lexical and discourse analysis of online chat dialog. In Semantic Computing, 2007. ICSC 2007. International Conference on, pages 19–26. IEEE. find
  7. Jonathan Herzig, Guy Feigenblat, Michal Shmueli-Scheuer, David Konopnicki, Anat Rafaeli, Daniel Altman, and David Spivak. 2016. Classifying emotions in customer support dialogues in social media. In SIGDIAL Conference, pages 64–73. find
  8. Minkyoung Kim and Harksoo Kim. 2018. Integrated neural network model for identifying speech acts, predicators, and sentiments of dialogue utterances. Pattern Recognition Letters. ↗
  9. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. Evaluation measures for the semeval-2016 task 4: Sentiment analysis in twitter (draft: Version 1.12). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. find
  10. Nicole Novielli and Carlo Strapparava. 2013. The role of affect analysis in dialogue act identification. IEEE Transactions on Affective Computing, pages 439–451. ↗
  11. Agnieszka Magdalena Pluwak. 2016. Towards application of speech act theory to opinion mining. Cognitive Studies, pages 33–44. ↗
  12. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 172–180. Association for Computational Linguistics. find
  13. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 553–562. find
  14. M. Thelwall, K. Buckley, and G. Paltoglou. 2012a. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, pages 163–173. ↗
  15. M. Thelwall, P. Sud, and F. Vis. 2012b. Commenting on YouTube videos: From Guatemalan rock to El Big Bang. Journal of the American Society for Information Science and Technology, pages 616–629. Soroush Vosoughi and Deb Roy. arXiv:1605.05156. 2016. Tweet acts: A speech act classifier for twitter. arXiv preprint Elina Zarisheva and Tatjana Scheffler. 2015. Dialog act annotation for twitter conversations. In SIGDIAL Conference, pages 114–123. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/cerisara-2018-multi-task-dialog-act-sentiment/