Elamur Research⌕
☰
home / papers / 2021 / BERT's The Word : Sarcasm Target Detection using BERT

BERT's The Word : Sarcasm Target Detection using BERT

Paper · 2021 · Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra et al.

Summary

This paper returns to sarcasm target detection: finding which words in a sarcastic text name the person, thing or situation being mocked. The 2019 ALTA shared task on this problem ended without a winner. The authors cast it as word-level sequence labelling. They adapt two aspect-based sentiment models from the ABSA-PyTorch repository, TD-BERT and BERT-AEN, so that each word is scored as a candidate target alongside the full sentence. For the Reddit and Twitter data, BERT Base first receives extra masked-language-model pre-training on in-domain text. Both models are then fine-tuned on three small corpora: Tweets, book snippets (Books) and the ALTA Reddit set. The domain-adapted TD-BERT does best everywhere. Measured by DICE, it is about 15.2% better than the strongest earlier system on Reddit, with smaller gains on Tweets and Books. On the ALTA Kaggle test portions it outperforms every original shared-task entry, but its margin over prior baselines is not statistically significant. The work offers a simple, reproducible transformer baseline for a task where the target may be missing from the text altogether.

Key points

  • Data:
    • Tweets: 224 sentences, 10% of targets OUTSIDE.
    • Books: 506 sentences, 5% OUTSIDE.
    • Reddit (ALTA 2019): 950 sentences, 35% OUTSIDE, meaning no target is present in the text.
  • Domain pre-training: extra MLM training, with 15% masking, drew on 150,000 posts taken from Khodak's Reddit corpus plus 100,000 tweets sampled from The Edinburgh Twitter Corpus. Sentences appearing in the evaluation data were removed first. Books received no extra pre-training.
  • Mean DICE with five-fold cross-validation, TD-BERT (PT) vs. the strongest earlier system (Baseline 2):
    • Reddit: 0.824 vs. 0.715
    • Tweets: 0.891 vs. 0.860
    • Books: 0.881 vs. 0.879 (plain TD-BERT)
    • The abstract and introduction quote slightly different improvement percentages.
  • Kaggle (ALTA 2019): TD-BERT (PT) scored 0.501 on the public portion and 0.562 on the private one. It beat the Powers and Orangutan entries and the Always OUTSIDE and Pronouns baselines at p < 0.05, but did not differ significantly from Baselines 1 and 2.
  • TD-BERT vs. BERT-AEN: the simpler TD-BERT was both faster and more accurate than the attention-heavy BERT-AEN. Extra pre-training took 336 minutes for Reddit and 240 minutes for Tweets.

Interesting ideas

  • I1. Candidate-word pairing: each candidate word goes in as a second segment next to the whole sentence, so the model judges one word in context rather than labelling the sequence blindly. This framing transfers to any task of the form "is this span the X of the sentence?"
  • I2. Explicit "no target" label: an OUTSIDE value is used when the mocked entity is not mentioned, which covers about a third of the Reddit data. This keeps the system from being forced to point at some word.
  • I3. Decontaminated domain adaptation: before further pre-training on large in-domain text, the authors removed every sentence that appears in the evaluation data. This lets them claim that gains come from register knowledge rather than leakage.
  • I4. Checking the pipeline itself: baseline reimplementations were confirmed against the published scores with ANOVA. The local DICE implementation was checked by uploading a run to a private Kaggle contest. Both steps verify the evaluation before trusting the results.
  • I5. Coverage analysis of a test drop: the weaker Kaggle score was explained by checking how many subreddits in the hidden split never appeared in training, which was 23%.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module in prag could score candidate tree nodes paired with their clause, which ties the predicted target of ridicule to a specific explainable dependency node.
  • I2 (frames): irony and implicature frames should allow an unfilled or implicit target slot instead of attaching to the nearest pronoun, which mirrors the weak Pronouns baseline.
  • I3 (learn): when building silver data or adapting gparse to new registers, filter out test-treebank sentences first so that later evaluations remain honest.
  • I4 (learn): new scorers and reimplemented comparison systems should first reproduce known reference numbers, as a sanity gate alongside negative controls.
  • I5 (learn): when accuracy falls on a new split, compute the overlap of genres, sources or vocabulary with the training data before blaming the model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 2 Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Sentiment & emotion · 608 works

Cite

Cite as Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra and David Eyers. 2021. BERT's The Word : Sarcasm Target Detection using BERT. Proceedings of the 19th Annual Workshop of the Australasian Language Technology Association.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{parameswaran2021bert,
  title = {{BERT's The Word : Sarcasm Target Detection using BERT}},
  author = {Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra and David Eyers},
  booktitle = {Proceedings of the 19th Annual Workshop of the Australasian Language Technology Association},
  year = {2021},
  url = {https://aclanthology.org/2021.alta-1.21/}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ameeta Agrawal and Aijun An. 2018. Affective representations for sarcasm detection. In 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pages 1029–1032. find
  2. Salvatore Attardo, Jodi Eisterhold, Jennifer Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. ↗
  3. Adithya Avvaru, Sanath Vobilisetty, and Radhika Mamidi. 2020. Detecting sarcasm in conversation context using transformer-based models. In Proceedings of the Second Workshop on Figurative Language Processing, pages 98–103. our card
  4. David Bamman and Noah A. Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of the 9th International Conference on Web and Social Media, ICWSM 2015, pages 574–577. find
  5. Arup Baruah, Kaushik Das, Ferdous Barbhuiya, and Kuntal Dey. 2020. Context-aware sarcasm detection using bert. In Proceedings of the Second Workshop on Figurative Language Processing, pages 83–87. our card
  6. Santosh Kumar Bharti, Korra Sathya Babu, and Sanjay Kumar Jena. 2015. Parsing-based sarcasm sentiment recognition in Twitter data. In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2015, pages 1373–1380. find
  7. Necva Bölücü and Burcu Can. 2020. Sarcasm target identification with lstm networks. In 2020 28th Signal Processing and Communications Applications Conference (SIU), pages 1–4. IEEE. find
  8. Henry S Cheang and Marc D Pell. 2008. The sound of sarcasm. Speech communication, 50(5):366–381. ↗
  9. Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, volume 1, pages 4171–4186. our card
  10. Christopher Ifeanyi Eke, Azah Anir Norman, Liyana Shuib, and Henry Friday Nweke. 2020. Sarcasm identification in textual data: systematic review, research challenges and open directions. Artificial Intelligence Review, 53(6):4215–4258. ↗
  11. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. CASCADE: Contextual Sarcasm Detection in Online Discussion Forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848. our card
  12. Mickel Hoang, Oskar Alija Bihorac, and Jacobo Rouces. 2019. Aspect-based sentiment analysis using bert. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 187–196. find
  13. Kevin Hylton. 2018. I’m not joking! the strategic use of humour in stories of racism. Ethnicities, 18(3):327–343. ↗
  14. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):1–22. ↗ our card
  15. Aditya Joshi, Pranav Goel, Pushpak Bhattacharyya, and Mark Carman. 2016a. Automatic Identification of Sarcasm Target: An Introductory Approach. ↗ our card
  16. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, Mark Carman, Meghna Singh, Jaya Saraswati, and Rajita Shukla. 2016b. How challenging is sarcasm versus irony classification?: A study with a dataset from English literature. In Proceedings of the Australasian Language Technology Association Workshop 2016, pages 123–127. our card
  17. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra-halli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579. ↗ our card
  18. Jennifer Ling and Roman Klinger. 2016. An empirical, quantitative analysis of the differences between sarcasm and Irony. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9989 LNCS:203–216. find
  19. Zhengjie Gao, Ao Feng, Xinyu Song, and Xi Wu. 2019. Target-dependent sentiment classification with bert. IEEE Access, 7:154290–154299. ↗
  20. Liyuan Liu, Jennifer Lewis Priestley, Yiyun Zhou, Herman E. Ray, and Meng Han. 2019. A2Text-Net: A Novel Deep Neural Network for Sarcasm Detection. In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), pages 118–126. Aniruddha Ghosh and Dr. find
  21. Tony Veale. 2016. Frack-ing Sarcasm using Neural Network. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 161–169. find
  22. Andrianarisoa Tojo Martini, Makhmudov Farrukh, and Hongwei Ge. 2018. Recognition of ironic sentences in Twitter using attention-based LSTM. International Journal of Advanced Computer Science and Applications, 9(8):7–11. ↗
  23. Debanjan Ghosh and Smaranda Muresan. 2018. “With 1 follower I must be AWESOME :P”. Exploring the role of irony markers in irony recognition. Icwsm, (Icwsm):588–591. our card
  24. Alice E Marwick and Danah Boyd. 2011. I tweet honestly, i tweet passionately: Twitter users, context collapse, and the imagined audience. New media & society, 13(1):114–133. ↗
  25. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620. ↗ our card
  26. Derek Miller. 2019. Leveraging bert for extractive text summarization on lectures. arXiv preprint arXiv:1906.04165. ↗
  27. Diego Molla and Aditya Joshi. 2019. Overview of the 2019 ALTA Shared Task : Sarcasm Target Identification. In Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association, pages 192–196. find
  28. Rajdeep Mukherjee, Shreyas Shetty, Subrata Chattopadhyay, Subhadeep Maji, Samik Datta, and Pawan Goyal. 2021. Reproducibility, replicability and beyond: Assessing production readiness of aspect based sentiment analysis in the wild. arXiv preprint arXiv:2101.09449. ↗
  29. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2021. Fighting hate speech, silencing drag queens? artificial intelligence in content moderation and risks to lgbtq voices online. Sexuality & Culture, 25(2):700–732. ↗
  30. Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra, and David Eyers. 2021. Detecting the target of sarcasm is hard: Really?? Information Processing & Management, 58(4):102599. ↗
  31. Jasabanta Patro, Srijan Bansal, and Animesh Mukherjee. 2019. A deep-learning framework to detect sarcasm targets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6335–6341. our card
  32. Saša Petrović, Miles Osborne, and Victor Lavrenko. 2010. The edinburgh twitter corpus. In Proceedings of the NAACL HLT 2010 workshop on computational linguistics in a world of social media, pages 25–26. find
  33. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309–17320. ↗ our card
  34. Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314. ↗
  35. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019a. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.09588. ↗ our card
  36. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019b. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194–206. Springer. ↗ our card
  37. Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232. ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parameswaran-2021-bert-s-word-sarcasm-target/