Elamur Research⌕
☰
home / papers / 2020 / TextCaps: a Dataset for Image Captioning with Reading Comprehension

TextCaps: a Dataset for Image Captioning with Reading Comprehension

Paper · 2020 · Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach et al.

Summary

Standard image captioning systems describe objects but ignore written text in a scene, even though such text often carries the key information, especially for blind and low-vision users. To address this, the authors define a task they call image captioning with reading comprehension and build TextCaps, a dataset of about 142k human-written captions for 28,408 Open Images v3 images. These are the same images used in TextVQA, all confirmed to contain text. Crowd workers wrote one-sentence descriptions that depend on reading that text, and a second group of workers voted on quality to filter them.

The captions require a model to:

  • choose which OCR tokens matter,
  • combine or paraphrase them, and
  • alternate repeatedly between vocabulary words and copied scene text, many of which never appear in training.

Conventional captioners such as BUTD essentially fail at this. An adaptation of the TextVQA model M4C, called M4C-Captioner, performs much better. Human captions still clearly outperform it, which leaves an open benchmark for future work.

Key points

  • Dataset and annotation: TextCaps contains 142,040 captions on 28,408 images (the abstract rounds this to 145k). Annotation had two stages. Writers produced captions that needed reading, then five evaluators judged each caption. The criteria were whether it required reading, was accurate, was one grammatical sentence, and was free of subjective language.
  • Why existing data falls short: Only about 2.7% of COCO captions mention scene text, and COCO's OCR vocabulary has fewer than 350 distinct tokens. Text-VQA answers are mostly 1–2 words, so they cannot teach full-sentence generation.
  • Model results: M4C-Captioner gave the best results. Ablations show it needs semantic, visual and spatial information about OCR tokens. Supplying ground-truth OCR narrows the gap to human performance but does not close it.
  • Transfer to COCO: Training jointly on COCO and TextCaps makes the model mention text in COCO images. Trained on COCO alone, it does not.
  • Availability: The dataset and its challenge server are hosted at textvqa.org/textcaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations🏢Meta (United Kingdom)London · United Kingdom54 works · 19,348 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksQuestion answering · 487 works

Cite

Cite as Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach and Amanpreet Singh. 2020. TextCaps: a Dataset for Image Captioning with Reading Comprehension. arXiv:2003.12462.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.12462 · arXiv 2003.12462

@misc{sidorov2020textcaps,
  title = {{TextCaps: a Dataset for Image Captioning with Reading Comprehension}},
  author = {Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach and Amanpreet Singh},
  year = {2020},
  doi = {10.48550/arxiv.2003.12462},
  eprint = {2003.12462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.12462}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal*, H., Desai*, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., Anderson, P.: nocaps: novel object captioning at scale. In: International Conference on Computer Vision (ICCV) (2019) 4 find
  2. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and recognition with embedded attributes. IEEE transactions on pattern analysis and machine intelligence 36(12), 2552–2566 (2014) 10 find
  3. Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: European Conference on Computer Vision. pp. 382– 398. Springer (2016) 10
  4. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6077–6086 (2018) 1, 2, 9, 10, 11 find
  5. Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., et al.: Vizwiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. pp. 333–342. ACM (2010)… find
  6. Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusiñol, M., Valveny, E., Jawahar, C., Karatzas, D.: Scene text visual question answering. arXiv preprint arXiv:1905.13648 (2019) 3, 4, 7 find
  7. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017) 10
  8. Borisyuk, F., Gordo, A., Sivakumar, V.: Rosetta: Large scale system for text detection and recognition in images. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 71–79. ACM (2018) 1, 4, 5, 7, 19 find
  9. Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015) 3, 7, 13
  10. Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 (2019) 1
  11. Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: Proceedings of the ninth workshop on statistical machine translation. pp. 376–380 (2014) 10
  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019) 1
  13. Goyal, P., Mahajan, D.K., Gupta, A., Misra, I.: Scaling and benchmarking self-supervised visual representation learning. International Conference on Computer Vision abs/1905.01235 (2019) 1 find
  14. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Vqa 2.0 evaluation. https://visualqa.org/evaluation.html 10 ↗
  15. Gurari, D., Zhao, Y., Zhang, M., Bhattacharya, N.: Captioning images taken by people who are blind. arXiv preprint arXiv:2002.08565 (2020) 4 — [arXiv:2002.08565](https://arxiv.org/abs/2002.08565) find
  16. He, T., Tian, Z., Huang, W., Shen, C., Qiao, Y., Sun, C.: An end-to-end textspotter with explicit alignment and attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5020–5029 (2018) 1, 4 O. Sidorov et al. find
  17. Hu, R., Singh, A., Darrell, T., Rohrbach, M.: Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. arXiv preprint arXiv:1911.06258 (2019) 3, 9, 10, 19 find
  18. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: IEEE International Conference on Computer Vision. pp. 4634–4643 (2019) 9, 11 find
  19. Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust reading competition. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1484–1493. IEEE (2013) 19 find
  20. Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., Murphy, K.: Openimages: A public dataset for large-scale multi-label and multi-c… find
  21. Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., Murphy, K.: Openimages: A public dataset for large-scale multi-label and multi-c… find
  22. Li, H., Wang, P., Shen, C.: Towards end-to-end text spotting with convolutional recurrent neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5238–5246 (2017) 1, 4 find
  23. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004) 10
  24. Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., Yan, J.: Fots: Fast oriented text spotting with a unified network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5676–5685 (2018) 1, 4 find
  25. Lu, J., Yang, J., Batra, D., Parikh, D.: Neural baby talk. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7219–7228 (2018) 9 find
  26. Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: ICDAR (2019) 3, 4, 7 find
  27. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: Neural Information Processing Systems (NIPS) (2011) 4, 7 find
  28. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics. pp. 311–318. Association for Computational Linguistics (2002) 10
  29. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015) 9, 10
  30. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565 (2018) 4, 7
  31. Singh, A., Goswami, V., Natarajan, V., Jiang, Y., Chen, X., Shah, M., Rohrbach, M., Batra, D., Parikh, D.: Mmf: A multimodal framework for vision and language research. — TextCaps: a Dataset for Image Captioning with Reading Comprehension https://github.com/facebookresearch/mmf (2020) 10 TextCaps: a Dataset for Image Captioning with Reading Comprehension ↗
  32. Singh, A., Natarajan, V., Jiang, Y., Chen, X., Shah, M., Rohrbach, M., Batra, D., Parikh, D.: Pythia-a platform for vision & language research. In: SysML Workshop, NeurIPS. vol. 2018 (2018) 10 find
  33. Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8317–8326 (2019) 3, 4, 5, 7, 22 find
  34. Smith, R.: An overview of the tesseract ocr engine. In: International Conference on Document Analysis and Recognition (ICDAR 2007). vol. 2, pp. 629–633. IEEE (2007) 1, 4 find
  35. Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015) 10
  36. Vinyals, O., Fortunato, M., Jaitly, N.: Pointer networks. In: Advances in Neural Information Processing Systems. pp. 2692–2700 (2015) 9 find
  37. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Glue: A multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of International Conference on Learning Representations (2019) 1
  38. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014) 3 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/