Elamur Research⌕
☰
home / papers / 2016 / How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text

How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text

Summary

Sarcasm labelling depends on cultural context as well as language skill. This raises the question of how reliable labels are when annotators come from a different culture than the text's authors. Joshi et al. examine this for one culture pair: Indian annotators labelling American English text. Two experienced Indian linguists re-annotated two existing sarcasm datasets, Tweet-A (tweets) and Discussion-A (forum posts), which had originally been labelled by American annotators.

The authors combine three analyses:

  • a qualitative look at where the Indian annotators struggled,
  • agreement statistics (Cohen's κ and percentage overlap),
  • the downstream effect on an SVM sarcasm classifier trained on each label set.

The Indian annotators agreed with each other more than with the American labels. Their main difficulties were unfamiliar named entities, such as sports figures and politically charged personalities, and US-specific social or political situations. Even so, classifiers trained on their labels performed only slightly worse, and the drop was not statistically significant. A further classifier predicted native/non-native disagreement from textual features with modest success. The study is small, but it offers an early framework for judging cross-cultural annotation quality, which is relevant whenever crowd-sourced annotators' backgrounds are unknown.

Key points

  • Data: Tweet-A (Riloff et al., 2013) has 2278 tweets, 506 of them sarcastic. Discussion-A (Walker et al., 2012; Amazon Mechanical Turk, US IPs) has 5854 posts, 742 of them sarcastic. Each item ends up with one American and two Indian labels, and the datasets are available on request.
  • Agreement on Tweet-A: κ was 0.686 between the two Indian annotators, versus 0.524 and 0.508 between each Indian annotator and the American labels. The original American annotators averaged 0.81.
  • Agreement on Discussion-A: κ was 0.700 between the Indian annotators, versus 0.569 and 0.288 with the American labels.
  • Classification (LibSVM, linear kernel, unigrams, 5-fold): AUC with American vs. Indian training labels was 0.816 vs. 0.771 on Tweet-A and 0.734 vs. 0.669 on Discussion-A. Neither drop was significant (p = 0.221 and 0.480).
  • Predicting disagreement (SVM with POS, named-entity and unigram features from NLTK, 3-fold): AUC was 0.56 on Tweet-A and 0.59 on Discussion-A, which is only slightly above chance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2111

@inproceedings{joshi2016how,
  title = {{How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text}},
  author = {Joshi, Aditya and Bhattacharyya, Pushpak and Carman, Mark and Saraswati, Jaya and Shukla, Rajita},
  booktitle = {Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities},
  year = {2016},
  doi = {10.18653/v1/w16-2111},
  url = {https://doi.org/10.18653/v1/w16-2111}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Patricia Rockwell and Evelyn M Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. ↗
  2. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the Annual Meeting of the Association for Computational Linguistics on Computational Linguistics 2006: Systems Demonstrations, pages 69–72. find
  3. John Schwoebel, Shelly Dews, Ellen Winner, and Kavitha Srinivas. 2000. Obligatory processing of the literal meaning of ironic utterances: Further evidence. Metaphor and Symbol, 15(1-2):47–61. ↗
  4. Frank Boers. 2003. Applied linguistics perspectives on cross-cultural variation in conceptual metaphor. Metaphor and Symbol, 18(4):231–238. ↗
  5. Deborah Tannen. 1984. The pragmatics of cross-cultural communication. Applied Linguistics, 5(3):189–195. find
  6. Lawrence F Bouton. 1988. A cross-cultural study of ability to interpret implicatures in english. World Englishes, 7(2):183–196. ↗
  7. Jenny Thomas. 1983. Cross-cultural pragmatic failure. ↗
  8. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27. find
  9. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In International AAAI Conference on Web and Social Media 2010. find
  10. Trevor Cohn and Lucia Specia. 2013. Modelling annotator bias with multi-task gaussian processes: An application to machine translation quality estimation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics on Computational Linguistics 2013, pages 32–42. find
  11. Marilyn A Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Language Resources and Evaluation Conference 2014, pages 812–817. find our card
  12. Amitava Das and Sivaji Bandyopadhyay. 2011. Dr sentiment knows everything! In Proceedings of the Annual Meeting of the Association for Computational Linguistics/Human Language Technologies 2011: Systems Demonstrations, pages 50–55. find
  13. Janyce M Wiebe, Rebecca F Bruce, and Thomas P O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proceedings of the Annual Meeting of the Association for Computational Linguistics on Computational Linguistics 1999, pages 246–253. find
  14. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning 2010, pages 107–116. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/joshi-2016-how-do-cultural-differences-impact/