Elamur Research⌕
☰
home / papers / 2012 / Towards Scalable Speech Act Recognition in Twitter: Tackling Insufficient Training Data

Towards Scalable Speech Act Recognition in Twitter: Tackling Insufficient Training Data

Summary

Automatically labeling tweets with speech act types is useful for studying user behaviour, but annotated tweets are scarce and expensive to produce. Building on their earlier supervised system (Zhang et al. 2011), the authors treat this as a five-class problem: statement, question, suggestion, comment and miscellaneous. They first plot learning curves on their existing datasets (News, Entity, LST and the combined Mixed set). These show that performance levels off well before all annotated data is used. They then simulate a realistic setting with only a small labeled seed (5% or 10% of the data) and many unlabeled tweets. In this setting they compare an inductive SVM baseline with two semi-supervised methods: transductive SVM and graph-based label propagation (GLP). Both semi-supervised methods benefit from the unlabeled data. Transductive SVM comes close to supervised performance despite far less training data. GLP performs worse because of the way it combines one-vs-all predictions. The study gives practical guidance for building speech act recognizers for Twitter at scale with little annotation effort.

Key points

  • The features come from the authors' prior work: speech-act cue words, special words (abbreviations, opinion words, vulgar words, emoticons) and Twitter-specific characters and punctuation. Text normalization is deliberately not applied.
  • Supervised learning curves reach a plateau at about 40% training data for News and Entity, 20% for Mixed and 10% for LST. Even 10% of Mixed still amounts to more than 800 hand-labeled tweets.
  • Doubling the labeled seed from 5% to 10% helps the semi-supervised methods only modestly, with a maximum gain of 15.9% (macro-F, Mixed, GLP). The inductive SVM, by contrast, improves by up to 41.3% (macro-F, Entity).
  • Confusion matrices on News, where 69% of tweets are statements, show the inductive SVM over-predicting statement. Transductive SVM moves many suggestion and comment tweets back to their correct classes. GLP assigns most tweets to statement or miscellaneous, because its one-vs-all scores are sensitive to differences between classes.
  • The authors conclude that class imbalance hurts as much as data scarcity. As future work, they plan to try other multi-class conversion schemes, natively multi-class learners (Naïve Bayes, Decision Tree) and unsupervised methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhang2012scalable,
  title = {{Towards Scalable Speech Act Recognition in Twitter: Tackling Insufficient Training Data}},
  author = {Zhang, Renxian and Gao, Dehong and Li, Wenjie},
  booktitle = {Proceedings of the Workshop on Semantic Analysis in Social Media},
  year = {2012},
  url = {https://aclanthology.org/W12-0603/}
}

References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Austin, J. 1962. How to Do Things with Words. Oxford: Oxford University Press. find
  2. Cohen, W., Carvalho, V., and Mitchell, T. 2004. Learning to Classify Email into “Speech Acts”. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP-04), 309–316. find our card
  3. Crystal, D. 2006. Language and the Internet, 2nd edition. Cambridge, UK: Cambridge University Press. find
  4. Crystal, D. 2011. Internet linguistics. London: Routledge. find
  5. Culp M. and Michailidis, G. 2007. An Iterative Algorithm for Extending Learners to a Semisupervised Setting. In The 2007 Joint Statistical Meetings (JSM). find
  6. Deshpande S. S., Palshikar, G. K., and Athiappan, G. 2010. An Unsupervised Approach to Sentence Classification, In International Conference on Management of Data (COMAD 2010), Nagpur, India. find
  7. Dhillon, R., Bhagat, S., Carvey, H., and Shriberg, E. 2004. Meeting Recorder Project: Dialog Act Labeling Guide. Technical report, International Computer Science Institute. ↗
  8. Erkan, G., Özgür, A., and Radev, D. 2007. Semi-Supervised Classification for Extracting Protein Interaction Sentences Using Dependency Parsing. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 228–237. Feng, D., Shaw, E., Kim, J., and Hovy. find
  9. E. H. 2006. Learning to Detect Conversation Focus of Threaded Discussions. In Proceedings of HLT-NAACL, 208–215. Haffari G.R. and Sarkar. find
  10. A. 2007. Analysis of semi-supervised learning with the Yarowsky algorithm. In 23rd Conference on Uncertainty in Artificial Intelligence (UAI). find
  11. Jeong, M., Lin, C-Y., and Lee, G. 2009. Semi-supervised Speech Act Recognition in Emails and Forums. In Proceedings of EMNLP, pages 1250– 1259. find
  12. Joachims, T. 1999. Transductive Inference for Text Classification using Support Vector Machines. In Proceedings of the 16th International Conference on Machine Learning (ICML). find
  13. Jurafsky, D., Shriberg, E., and Biasca, D. 1997. Switchboard SWBD-DAMSL Labeling Project Coder’s Manual, Draft 13. Technical report, University of Colorado Institute of Cognitive Science. find
  14. Medlock, B., and Briscoe, T. 2007. Weakly Supervised Learning for Hedge Classification in Scientific Literature. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 992–999. find our card
  15. Searle, J. 1975. Indirect speech acts. In P. Cole and J. Morgan (eds.), Syntax and semantics, vol. iii: Speech acts (pp. 59–82). New York: Academic Press. ↗
  16. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R. Van Ess-Dykema, C., and Meteer, M. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3):339–373. ↗ our card
  17. Vapnik, V. 1998. Statistical Learning Theory. New York: John Wiley & Sons. find
  18. Yarowsky, D. 1995. Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL-1995), 189–196. find our card
  19. Zhang, R., Gao, D., and Li, W. 2011. What Are Tweeters Doing: Recognizing Speech Acts in Twitter. In AAAI-11 Workshop on Analyzing Microtext. find
  20. Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Scholkopf, B. 2004. Learning with Local and Global Consistency. Advances in Neural Information Processing Systems (NIPS), vol. 16, Cambridge, MA: MIT Press. find
  21. Zhu, X., Ghahramani, Z., and Lafferty, J. D. 2003. Semi-supervised Learning Using Gaussian Fields and Harmonic Functions. In Proceedings of the Twentieth International Conference on Machine Learning (ICML), 912–919, Washington, DC. find
  22. Zhu, X. and Goldberg, A. B., 2009. Introduction to Semi-Supervised Learning. Morgan & Claypool Publishers. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zhang-2012-scalable-speech-act-recognition-twitter/