📄 Sarcasm Identification and Detection in Conversion Context using BERT

Summary

This paper describes a system built for the FigLang 2020 shared task, which challenges participants to identify sarcasm within conversations taking place on social media. Each instance is a reply (response) to a dialogue thread (context), and the goal is to label it SARCASM or NOT_SARCASM. The authors also examine how much of the preceding conversation actually helps the decision.

They work with the organisers' Twitter and Reddit datasets. The pipeline cleans social-media noise: user mentions, URLs, hashtags, elongated words, censored profanity, contractions and run-together words. Three families of models are then compared:

• classical classifiers on Doc2Vec or TF-IDF features;
• an RNN-LSTM with Word2Vec embeddings;
• a fine-tuned uncased BERT model.

Each is fed different combinations of context and response.

BERT trained on the response alone gives the best scores on both forums. Adding the conversation context did not improve results. The LSTM struggled on such small training sets. For anyone building sarcasm or offensive-language classifiers, the takeaway is practical: with little data, a pretrained transformer on the target utterance is a strong baseline, and longer dialogue context is not automatically useful.

Key points

• Data: Twitter train has 5,000 balanced instances (2,500 per class), and Reddit train has 4,400 (2,200 per class). Each test set contains 1,800 items.
• Best results: the system scored macro-F1 0.722 on Twitter and 0.679 on Reddit, both with BERT on the isolated response (R). On Twitter, combining context with the response (CR) dropped F1 to 0.671.
• Classical baselines: LR, RF, XGBoost, linear SVC and Gaussian NB were tested. TF-IDF features clearly beat Doc2Vec, which stayed near chance (about 0.50 cross-validation accuracy). The authors attribute this to ungrammatical text.
• LSTM: the RNN-LSTM performed poorly (F1 0.361 on Twitter CR), which the authors blame on the small dataset size.
• BERT setup: uncased model with batch size 32, learning rate 2e-5, 3 epochs and warmup proportion 0.1.