Elamur Research⌕
☰
home / papers / 2020 / Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context

Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context

Summary

The paper addresses sarcasm detection in dialogue: deciding whether a reply is sarcastic given the preceding turns. The authors entered the FigLang2020 shared task, which provided small labeled Twitter and Reddit sets, and made two main contributions.

The first is Contextual Response Augmentation (CRA), which turns cheap unlabeled conversation threads into labeled examples. For each unlabeled reply, CRA retrieves similar labeled replies and uses BERT next-sentence-prediction scores to pick the one that best fits the thread. That reply and its label are then attached to the unlabeled context. Labeled data is expanded as well: context-only sequences become negative examples, and back-translation produces extra positives.

The second is a context ensemble. Models are trained on several context window sizes, which evens out the mix of short and long threads.

The classifier is BERT-large with BiLSTM and NeXtVLAD pooling. As team miroblog, the system placed first on both datasets by clear margins. This shows that unlabeled dialogue can meaningfully improve figurative-language classifiers.

Key points

  • Results and data: The system ranked 1st of 35 teams on Twitter (F1 0.931) and 1st of 34 on Reddit (F1 0.834).
    • Twitter split: 4000 train / 1000 validation / 1800 test.
    • Reddit split: 3520 train / 880 validation / 1800 test.
  • CRA pipeline:
    • Sentence-BERT encodes the replies.
    • The top 1000 similar labeled replies are retrieved.
    • A fine-tuned BERT NSP model reranks them.
    • The label of the best candidate is transferred to the unlabeled thread.
    • About 100,000 crawled Twitter/Reddit texts served as the unlabeled source.
  • Additional corpora: Ptáček/Platek (57,041), Riloff (1,570), SARC-v2 (321,748), SARC-v2-pol (14,340) and SemEval-2018-irony (3,851). Back-translation for positives used French, Spanish and Dutch.
  • Augmentation ablation (Twitter validation F1):
    • No augmentation: 0.854.
    • Labeled augmentation: 0.861.
    • Adding unlabeled CRA: 0.897.
  • Architecture and context findings:
    • NeXtVLAD beat max and mean pooling. After augmentation it reached F1 0.898 vs 0.853 on Twitter and 0.751 vs 0.721 on Reddit (vs max pooling).
    • An ensemble with maximum context outperformed one limited to 3 turns by only about 0.006 F1. This suggests that turns older than the last three add little.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.2 · arXiv 2006.06259

@inproceedings{lee2020augmenting,
  title = {{Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context}},
  author = {Lee, Hankyol and Yu, Youngjae and Kim, Gunhee},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.2},
  eprint = {2006.06259},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.2}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. 2016. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297–5307. find
  2. Alexandre Bérard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs europe’s systems for the wmt19 machine translation robustness task. In arXiv preprint arXiv:1907.06488. ↗
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  4. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579. ↗ our card
  5. Rongcheng Lin, Jing Xiao, and Jianping Fan. 2018. Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0. find
  6. Eugenio Martı́nez-Cámara, M Teresa Martı́n-Valdivia, L Alfonso Urena-López, and A Rturo Montejo-Ráez. 2014. Sentiment analysis in twitter. Natural Language Engineering, 20(1):1–28. find
  7. Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu. 2016. Sentiment analysis in social networks. Morgan Kaufmann. ↗
  8. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. find our card
  9. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  10. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card
  11. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681. ↗
  12. Leslie N Smith. 2017. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472. IEEE. find
  13. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50. find our card
  14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  15. Renjie Zheng, Hairong Liu, Mingbo Ma, Baigong Zheng, and Liang Huang. 2019. Robust machine translation with domain sensitive pseudo-sources: Baidu-osu wmt19 mt robustness shared task system report. arXiv preprint arXiv:1906.08393. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lee-2020-augmenting-data-sarcasm-detection-unlabeled/