Elamur Research⌕
☰
home / papers / 2020 / Ciron: a New Benchmark Dataset for Chinese Irony Detection

Ciron: a New Benchmark Dataset for Chinese Irony Detection

Summary

Work on irony detection has focused mostly on English, and Chinese has lacked labeled data suitable for training machine learning systems. Earlier Chinese resources, such as the NTU Irony Corpus, were built by searching for predefined lexical or syntactic cues. That approach limits coverage to those patterns, and in the NTU case it produced only positive examples. The authors address this gap with Ciron, a corpus of about 8.7K Weibo microblog posts collected without pattern-based preselection, so it covers a broader range of ironic and non-ironic language. Five native-speaker postgraduate annotators rated every post on a five-level scale, from not ironic to strongly ironic. They followed the reversal-based Irony Identification Procedure (IIP), and their agreement was moderate. To show the resource is usable, the team trained seven baseline classifiers, from classical models to neural ones including BERT. The paper presents Ciron as the first sizable, finely graded gold standard for Chinese irony detection. Such a resource can support downstream work like affective analysis and hate speech detection.

Key points

  • Ciron contains about 8.7K posts from Weibo with private user data removed, and it is released at github.com/Christainx/Ciron.
  • Each post received labels from five annotators aged 24–30, all native Chinese speakers, using a 5-point scale where 1 marks a post without irony, 2 marks irony as improbable, 3 means there is too little evidence to decide, 4 indicates weak irony, and 5 indicates strong irony.
  • Annotation follows the IIP of Li et al. (2019), which treats irony as a reversal between literal and contextual meaning, and inter-annotator agreement reached a Fleiss' Kappa of 0.470.
  • Baselines evaluated on the dataset are NB, LR, SVM, CNN, LSTM, BiLSTM-AT and BERT.
  • For comparison, the earlier NTU Irony Corpus (Traditional Chinese, Yahoo Blog) holds 1,005 pattern-selected ironic posts and no negative examples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xiang2020ciron,
  title = {{Ciron: a New Benchmark Dataset for Chinese Irony Detection}},
  author = {Xiang, Rong and Gao, Xuefeng and Long, Yunfei and Li, Anran and Chersoni, Emmanuele and Lu, Qin and Huang, Chu-Ren},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.701/}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Apter, M. J. (2007). Reversal Theory: The Dynamics of Motivation, Emotion, and Personality. find
  2. Oneworld Publications Limited. Barbieri, F. and Saggion, H. (2014). Automatic detection of irony and humour in twitter. In ICCC, pages 155–162. find
  3. Buschmeier, K., Cimiano, P., and Klinger, R. (2014). An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49. find our card
  4. Carvalho, P., Sarmento, L., Silva, M. J., and De Oliveira, E. (2009). Clues for Detecting Irony in User-Generated Contents: Oh...!! It’s so Easy;-. In Proceedings of the 1st International CIKM Workshop on Topic-sentiment Analysis for Mass Opinion, pages 53–56. find
  5. ACM. Deng, Z., Jia, X., and Chen, J. (2015). Research on Chinese Irony Detection in Microblog. In Computer Engineering Science, volume 37(12), pages 2312– 2317. find
  6. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. find
  7. Ebert, S., Vu, N. T., and Schütze, H. (2015). A Linguistically-Informed Convolutional Neural Network. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 109–114. find
  8. Farías, D. I. H., Patti, V., and Rosso, P. (2016). Irony Detection in Twitter: The Role of Affective Content. ACM Transactions on Internet Technology (TOIT), 16(3):19. find
  9. Ghosh, A. and Veale, T. (2016). Fracking Sarcasm Using Neural Network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169. find our card
  10. Ghosh, A., Li, G., Veale, T., Rosso, P., Shutova, E., Barnden, J., and Reyes, A. (2015). Semeval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of Semeval, pages 470– 478. find
  11. Ghosh, D., Musi, E., Upasani, K., and Muresan, S. (2019). Interpreting verbal irony: Linguistic strategies and the connection to thetype of semantic incongruity. arXiv preprint arXiv:1911.00891. find our card
  12. Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  13. Huang, Y.-H., Huang, H.-H., and Chen, H.-H. (2017). Irony Detection with Attentive Recurrent Neural Networks. In Proceedings of ECIR, pages 534–540. find
  14. Springer. Huang, C.-R. (2019). Double Meaning and Reversal: Toward an Empirical Linguistic Account of Irony. In In 2019 Joint Conference of Linguistic Societies in Korea The 26th Joint Workshop on Linguistics and Language Processing. find
  15. Jia, X., Deng, Z., Min, F., and Liu, D. (2019). Three-Way Decisions Based Feature Fusion for Chinese Irony Detection. International Journal of Approximate Reasoning, 113:324–335. find
  16. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2017). Automatic Sarcasm Detection: A Survey. ACM Computing Surveys (CSUR), 50(5):73. find our card
  17. Karoui, J., Benamara, F., and Moriceau, V. (2019). Automatic Detection of Irony: Opinion Mining in Microblogs and Social Media. find
  18. John Wiley & Sons. Khattri, A., Joshi, A., Bhattacharyya, P., and Carman, M. (2015). Your Sentiment Precedes You: Using an Author s Historical Tweets to Predict Sarcasm. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25–30. find
  19. Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. arXiv preprint arXiv:1408.5882. find our card
  20. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324. find
  21. Li, A.-r. and Huang, C.-R. (2019). A Method of Modern Chinese Irony Detection. In The 19th Chinese Lexical Semantics Workshop, pages 273–288. find
  22. Li, A.-R., Chersoni, E., Xiang, R., Huang, C.-R., and Lu, Q. (2019). On the Easy Task of Evaluating Chinese Irony Detection. In Proceedings of PACLIC. find
  23. Long, Y., Xiang, R., Lu, Q., Huang, C.-R., and Li, M. (2019). Improving Attention Model Based on Cognition Grounded Data for Sentiment Analysis. IEEE Transactions on Affective Computing. find
  24. Maynard, D. and Greenwood, M. A. (2014). Who Cares About Sarcastic Tweets? Investigating the Impact of Sarcasm on Sentiment Analysis. In Proceedings of LREC. find our card
  25. ELRA. Pennington, J., Socher, R., and Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. In Proceedings of EMNLP, pages 1532–1543. find
  26. Reyes, A. and Rosso, P. (2014). On the Difficulty of Automatically Detecting Irony: Beyond a Simple Case of Negation. Knowledge and Information Systems, 40(3):595–614. find
  27. Tang, Y.-j. and Chen, H.-H. (2014). Chinese Irony Corpus Construction and Ironic Structure Analysis. In Proceedings of COLING, pages 1269–1278. find our card
  28. Tang, D., Qin, B., and Liu, T. (2015). Learning Semantic Representations of Users and Products for Document Level Sentiment Classification. In Proceedings of ACL. find
  29. Van Hee, C., Lefever, E., and Hoste, V. (2018). Semeval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of Semeval, pages 39–50. Association for Computational Linguistics. find our card
  30. Van Hee, C. (2017). Can Machines Sense Irony? Exploring Automatic Irony Detection on Social Media. Ph.D. thesis, Ghent University. find
  31. Wu, C., Wu, F., Wu, S., Liu, J., Yuan, Z., and Huang, Y. (2018). Thu_ngn at Semeval-2018 Task 3: Tweet Irony Detection with Densely Connected Lstm and Multi-Task Learning. In Proceedings of Semeval, pages 51–56. find
  32. Xing, F. Z. and Xu, Y. (2015). A Logistic Regression Model of Irony Detection in Chinese Internet Texts. Research in Computing Science, 90:239–249. find
  33. Zhang, Y., Wang, J., and Zhang, X. (2018). YNU-HPCC at SemEval-2018 Task 1: BiLSTM with Attention based Sentiment Analysis for Affect in Tweets. In Proceedings of Semeval, pages 273–278. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiang-2020-ciron-new-benchmark-dataset-chinese/