Elamur Research⌕
☰
home / papers / 2018 / Visual Attention Model for Name Tagging in Multimodal Social Media

Visual Attention Model for Name Tagging in Multimodal Social Media

Paper · 2018 · Di Lu, Leonardo Neves, Vitor Carvalho et al.

Summary

The paper addresses name tagging in social media posts that pair an image with a short, noisy caption. In such posts the text alone is often ambiguous: "Modern Baseball" could be a band or a reference to the sport. The authors built two new multimodal benchmarks, one from Twitter and one from crowdsourced Snapchat public stories. They extended the BLSTM-CRF tagger of Lample et al. (2016) and Ma and Hovy (2016) with two additions. A visual attention module uses an LSTM-encoded sentence query to weight ResNet region features, producing a visual context vector. A modulation gate then decides, word by word, how much visual information to combine with the textual features before the CRF layer. The model is trained end to end from image–sentence pairs without hand-crafted features. It beats state-of-the-art text-only taggers by roughly 3–4 absolute F-score points. The attention maps also make the model's decisions easier to interpret. The work shows that images can supply the context missing from short social media text.

Key points

  • Datasets: The authors created two new name tagging datasets, one from Twitter (using archive.org twitterstream data) and one from Snapchat captions. The Twitter annotations are offered to researchers on request.
  • Architecture:
    • Word embeddings are combined with character-level BLSTM representations.
    • These feed a BLSTM-CRF tagger whose LSTM is initialized with the attention-based visual context vector.
    • A modulation gate controls how visual and textual features are fused for each word.
  • Visual features: Images are rescaled to 224x224 and passed through ResNet. The last convolutional layer gives 7x7 region vectors of dimension 1,024, each covering a 32x32 pixel patch. A 1,024-dimensional global vector serves as a comparison baseline.
  • Attention choice: The authors chose "linear projection after summation" over dot-product attention. It spreads attention more widely across related regions, whereas dot-product attention concentrates it and may bias the model.
  • Results and errors: The model achieves a 3–4% absolute F-score gain over the baselines. Remaining errors come from three sources: images unrelated to the text, blurry images, and attention focused on the wrong region.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rensselaer Polytechnic InstituteTroy · United States11 works · 4,917 citations🏢SnapLos Angeles · United States4 works · 7,404 citations🏢IntuitMountain View · United States4 works · 887 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 269 worksNamed entity recognition · 101 works

Cite

Cite as Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang and Heng Ji. 2018. Visual Attention Model for Name Tagging in Multimodal Social Media. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1185

@inproceedings{lu2018visual,
  title = {{Visual Attention Model for Name Tagging in Multimodal Social Media}},
  author = {Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang and Heng Ji},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1185},
  url = {https://doi.org/10.18653/v1/p18-1185}
}

References

39 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In Proceedings of the 3rd Workshop on Noisy User-generated Text. find
  2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2017. Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998. — [arXiv:1707.07998](https://arxiv.org/abs/1707.07998) find
  3. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proceedings of the 2015 International Conference on Learning Representations.
  5. Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings of the Workshop on Noisy User-generated Text.
  6. Rich Caruana, Steve Lawrence, and C Lee Giles. 2001. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Proceedings of the 2001 Advances in Neural Information Processing Systems. find
  7. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960. — [arXiv:1511.05960](https://arxiv.org/abs/1511.05960) find
  8. Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of the 19th international conference on Computational Linguistics. find
  9. Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association of Computational Linguistics.
  10. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural…
  11. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL.
  12. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the 2013 IEEE international conference on acoustics, speech and signal processing. find
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. find
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  15. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning. find
  16. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  17. Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation. find
  18. Nut Limsopatham and Nigel Henry Collier. 2016. Bidirectional lstm for named entity recognition in twitter messages. In Proceedings of the 2nd Workshop on Noisy User-generated Text. find
  19. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.
  20. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proceedings of the 2014 European Conference on Computer Vision.
  21. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Proceedings of the 2016 Advances In Neural Information Processing Systems. find
  22. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  23. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  24. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  25. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
  26. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Lingui…
  27. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning.
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing.
  29. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the conference on Empirical Methods in Natural Language Processing.
  30. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision.
  31. Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics.
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 2017 Advances in Neural Information Processing Systems.
  33. Huijuan Xu and Kate Saenko. 2016. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In Proceedings of the 2016 European Conference on Computer Vision. find
  34. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 2015 International Conference on Machine Learning.
  35. Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W Cohen, and Ruslan Salakhutdinov. 2016a. Words or characters? fine-grained gating for reading comprehension. In Proceedings of the 2016 International Conference on Learning Representations. find
  36. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016b. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
  37. Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016. Video paragraph captioning using hierarchical recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. find
  38. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  39. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2018-visual-attention-model-name-tagging/