Elamur Research⌕
☰
home / papers / 2023 / Dynamic Routing Transformer Network for Multimodal Sarcasm Detection

Dynamic Routing Transformer Network for Multimodal Sarcasm Detection

Summary

The paper tackles multimodal sarcasm detection, where sarcasm in an image-text post is usually signalled by a mismatch between the two modalities. The authors argue that earlier systems capture this mismatch with fixed architectures, so they cannot adapt to the variety of sarcastic pairs. Some posts clash on local details, such as an object versus a phrase. Others clash in overall mood. They propose DynRT-Net, a routing-based dynamic network. Text is encoded with RoBERTa and images with a ViT. A stack of Dynamic Routing Transformer (DynRT) layers follows, and each layer contains a router. The router assigns input-dependent weights to several co-attention variants between text tokens and image patches, and the number of variants grows layer by layer (hierarchical co-attention). TRAR routes only within the image modality, whereas DynRT routes on the cross-modal co-attention itself. On the single public Twitter-based benchmark, the model is reported to beat previous state-of-the-art methods. The work matters because it applies dynamic network design to cross-modal incongruity for the first time.

Key points

  • Architecture: RoBERTa text encoder plus ViT image encoder, followed by stacked DynRT layers and a classifier that combines routed features with image features. Code is available at github.com/TIAN-viola/DynRT.
  • Routing works over at most four types of co-attention mask matrices. Increasing the number of mask types in deeper layers outperformed variants with equal or decreasing types.
  • In ablations, every substitute performed worse than DynRT. The substitutes were TRAR, a standard multimodal transformer layer, plain concatenation of the encoder [CLS] vectors (a sharp drop), and averaged or fixed attention in place of router predictions.
  • Depth varied from 1 to 6 layers; gains peaked early and 4 DynRT layers were chosen as the most stable and best configuration.
  • Limitations: only one public dataset exists, which restricts generalization testing, and only four co-attention designs were explored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.139

@inproceedings{tian2023dynamic,
  title = {{Dynamic Routing Transformer Network for Multimodal Sarcasm Detection}},
  author = {Tian, Yuan and Xu, Nan and Zhang, Ruike and Mao, Wenji},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.139},
  url = {https://doi.org/10.18653/v1/2023.acl-long.139}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. Computing Research Repository, arXiv:1607.06450. ↗ our card
  2. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2506–2515. find our card
  3. Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. 2017. Modulating early visual processing by language. In Proceedings of the International Conference on Neural Information Processing Systems, pages 6597–6607. ↗
  4. Jacob Devlin, Mingwei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186. find
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, pages 1–22. ↗
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770– 778. ↗
  7. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):1–22. ↗ our card
  8. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1746–1751. find our card
  9. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, pages 1–15. ↗
  10. Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. 2021. Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In Proceedings of the ACM International Conference on Multimedia, pages 4707–4715. find
  11. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022. Multimodal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1767–1777. find our card
  12. Hui Liu, Wenya Wang, and Haoliang Li. 2022. Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4995–5006. find our card
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pre-training approach. Computing Research Repository, arXiv:1907.11692. ↗
  14. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1383–1392. find our card
  15. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3942–3951. ↗ our card
  16. Shraman Pramanick, Aniket Roy, and Vishal M. Patel Johns. 2022. Multimodal learning using optimal transport for sarcasm and humor detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Visio, pages 546–556. find
  17. Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. 2021. Dynamic modality interaction modeling for image-text retrieval. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1104–1113. find
  18. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card
  19. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the ACM International Conference on Multimedia, pages 1136–1145. find our card
  20. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1010–1020. find our card
  21. Joseph Tepperman, David Traum, and Shrikanth Narayanan. 2006. "Yeah right": Sarcasm recognition for spoken dialogue systems. In Proceedings of the International Conference on Spoken Language Processing, pages 1838–1841. find
  22. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proceedings of the International AAAI Conference on Weblogs and Social Media, pages 162– 169. find

↑ back to top