Elamur Research⌕
☰
home / papers / 2015 / That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets

That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets

Summary

The paper argues that behavioral science could use social media as a cheap, geographically diverse alternative to lab studies. It tests this with tweets about annoying behaviors. Drawing on the geotagged collection assembled by Cheng et al. (2010), which contains 9 million tweets, the authors pulled out 3,375 posts tagged #petpeeve. Using an LDA-assisted, two-pass human annotation scheme, they labeled each post with one of 60 fine-grained categories.

For qualitative exploration, they used the supervised SAGE topic model to surface characteristic words for each category and for each US state.

For the prediction task, they built a 60-way logistic regression classifier with several feature types:

  • unigrams
  • part-of-speech features
  • dependency triples from MaltParser
  • frame-semantic features from SEMAFOR

Their main technical idea is data augmentation. Instead of feeding embeddings in as features, they replace words with their nearest neighbors in Word2Vec spaces (lexical and frame-semantic) to generate extra training tweets. Frame features and augmentation both yielded significant gains. This suggests external embeddings can offset the scarce, noisy data typical of short-text classification.

Key points

  • Dataset: 3,375 #petpeeve tweets from a corpus covering 121K users, released publicly. There are 60 categories; appearance is the largest at 14%, and 17% of tweets are unrelated. Inter-annotator agreement was moderate (0.445). Experiments use a 60/40 train/test split.
  • Feature comparison (F1): lexical 0.341, +POS 0.346, +dependencies 0.350, +semantic frames 0.366.
  • Augmentation setup: k = 5 nearest neighbors, which expands the training set fivefold. Lexical embeddings come from GoogleNews (100B words), Twitter (51M words), and Urban Dictionary (UD, 53M words). Frame embeddings come from Twitter (27M frames).
  • Lexical-only augmentation: GoogleNews embeddings raise F1 to 0.365, a 6.1% relative gain. The best overall F1, 0.380, combines UD lexical embeddings with frame-semantic embeddings on top of all features, a 3.8% relative gain.
  • Qualitative findings: SAGE surfaces regional patterns, for example "dmv" in MD and DC, linked to the 2010 snowstorm. The classifier is LibShortText logistic regression, with significance assessed by a paired t-test.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

79 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1306

@inproceedings{wang2015that,
  title = {{That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using \#petpeeve Tweets}},
  author = {Wang, William Yang and Yang, Diyi},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1306},
  url = {https://doi.org/10.18653/v1/d15-1306}
}

References

44 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the Workshop on Languages in Social Media, pages 30–38. Association for Computational Linguistics. find
  2. Sileye O Ba and Jean-Marc Odobez. 2011. Mul-tiperson visual focus of attention from head pose and meeting contextual cues. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(1):101–116. find
  3. Ryan SJD Baker and Kalina Yacef. 2009. The state of educational data mining in 2009: A review and future visions. JEDM-Journal of Educational Data Mining, 1(1):3–17. find
  4. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. find our card
  5. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media. find
  6. Marian Bartlett, Gwen Littlewort, Tingfan Wu, and Javier Movellan. 2008. Computer expression recognition toolbox. In Automatic Face & Gesture Recognition, 2008. FG’08. 8th IEEE International Conference on, pages 1–2. IEEE. find
  7. Sumit Basu. 2002. Conversational scene analysis. Ph.D. thesis, MaSSachuSettS InStitute of Technology. find
  8. Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of Computational Science, 2(1):1–8. find
  9. Jake Y Chen and Stefano Lonardi. 2009. Biological data mining. CRC Press. find
  10. Zhiyuan Cheng, James Caverlee, and Kyumin Lee. 2010. You are where you tweet: a content-based approach to geo-locating twitter users. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 759– 768. ACM. find
  11. D. Das, N. Schneider, D. Chen, and N.A. Smith. 2010. Probabilistic frame-semantic parsing. In HLT-NAACL 2010, page 948956, Los Angeles, California, USA, June. find our card
  12. Richard De Charms. 2013. Personal causation: The internal affective determinants of behavior. Routledge. find
  13. Jacob Eisenstein, Amr Ahmed, and Eric P Xing. 2011. Sparse additive generative models of text. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1041–1048. find
  14. Kenneth Alberto Funes Mora and J Odobez. 2012. Gaze estimation from multimodal kinect data. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on, pages 25–30. IEEE. find
  15. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Associ… find
  16. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 581–586. Association for Computa… find our card
  17. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A Smith. 2014. A dependency parser for tweets. In EMNLP. find our card
  18. David Lazer, Alex Sandy Pentland, Lada Adamic, Sinan Aral, Albert Laszlo Barabasi, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, et al. 2009. Life in the network: the coming age of computational social science. Science (New York, NY), 323(5915):721. find
  19. Omer Levy and Yoav Goldberg. 2014. Dependencybased word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2, pages 302–308. find
  20. Jiwei Li, Alan Ritter, Claire Cardie, and Eduard Hovy. 2014a. Major life event extraction from twitter based on congratulations/condolences speech acts. In Proceedings of Empirical Methods in Natural Language Processing. find
  21. Jiwei Li, Xun Wang, and Eduard Hovy. 2014b. What a nasty day: Exploring mood-weather relationship from twitter. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 1309– 1318. ACM. find
  22. Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 176–186, Sofia, Bulgaria, August. Association for Computational Linguis… find
  23. Alvaro Marcos-Ramiro, Daniel Pizarro-Perez, Marta Marron-Romera, Laurent Nguyen, and Daniel Gatica-Perez. 2013. Body communicative cue extraction for conversational analysis. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on, pages 1–8. IEEE. find
  24. Manuel Jesús Marin-Jimenez, Andrew Zisserman, Marcin Eichner, and Vittorio Ferrari. 2014. Detecting people looking at each other in videos. International Journal of Computer Vision, 106(3):282–296. find
  25. Marianne Schmid Mast, Daniel Gatica-Perez, Denise Frauendorfer, Laurent Nguyen, and Tanzeem Choudhury. 2015. Social sensing for psychology automated interpersonal behavior assessment. Current Directions in Psychological Science, 24(2):154–160. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  28. Greg Miller. 2011. Social scientists wade into the tweet stream. Science, 333(6051):1814–1815. find
  29. Laurent Nguyen, Jean-Marc Odobez, and Daniel Gatica-Perez. 2012. Using self-context for multimodal detection of head nods in face-to-face interactions. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 289–292. ACM. find
  30. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(02):95–135. find
  31. Brendan O’Connor, Ramnath Balasubramanyan, Bryan R Routledge, and Noah A Smith. 2010a. From tweets to polls: Linking text sentiment to public opinion time series. ICWSM, 11:122–129. find
  32. Brendan O’Connor, Michel Krieger, and David Ahn. 2010b. Tweetmotif: Exploratory search and topic summarization for twitter. In ICWSM. find
  33. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Proc of NAACL. find
  34. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1524–1534. Association for Computational Linguistics. find
  35. Alan Ritter, Oren Etzioni, Sam Clark, et al. 2012. Open domain event extraction from twitter. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1104–1112. ACM. find
  36. Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM, 56(1):116–124. find
  37. Yanchuan Sim, Noah A. Smith, and David A. Smith. 2012. Discovering factions in the computational linguistics community. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, ACL ’12 Special Workshop on Rediscovering 50 Years of Discoveries. find
  38. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume… find
  39. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics. find our card
  40. William Yang Wang, Elijah Mayfield, Suresh Naidu, and Jeremiah Dittmar. 2012. Historical analysis of legal opinions with a sparse mixed-effects latent variable model. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 740–749. Asso… find
  41. Shiliang Wang, Michael J Paul, and Mark Dredze. 2015. Social media as a sensor of air quality and public response in china. Journal of medical Internet research, 17(3). find
  42. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics (TACL), 2(1). find
  43. H Yu, C Ho, Y Juan, and C Lin. 2013. Libshorttext: A library for short-text classification and analysis. Technical report, Technical Report. http://www.csie. ntu. edu. tw/˜ cjlin/ papers/libshorttext. pdf. ↗
  44. Ni Zhang, Shelly Campo, Kathleen F Janz, Petya Eck-ler, Jingzhen Yang, Linda G Snetselaar, and Alessio Signorini. 2013. Electronic word of mouth on twitter about physical activity in the united states: exploratory infodemiology study. Journal of medical Internet research, 15(11). find

↑ back to top

This page in Markdown →