Elamur Research
☰
home / papers / 2016 / A Diagram Is Worth A Dozen Images

A Diagram Is Worth A Dozen Images

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1603.07396

Summary

The paper addresses how to get machines to understand diagrams. Diagrams, such as school science illustrations, show processes, cycles and dependencies that photographs cannot easily capture, but computer vision had paid them little attention. The authors split the problem into two tasks. The first is syntactic parsing: finding a diagram's elements and how they are structurally linked. The second is semantic interpretation, which they test through multiple-choice question answering. They encode a diagram's structure as a Diagram Parse Graph (DPG), building on Engelhardt's theory of graphic representation. Dsdp-Net, an LSTM model, builds a DPG step by step by choosing relationships from noisy candidate detections. Dqa-Net answers questions by attending to the relevant relations in the DPG. Both models are evaluated on AI2D, a new dataset of annotated science diagrams with questions. Dsdp-Net beats greedy and A* search baselines at parsing. Dqa-Net beats a standard VQA model that uses only image features. The work established diagram understanding as its own research area and released a benchmark for it.

Key points

  • AI2D dataset: more than 5,000 grade-school science diagrams, over 150,000 annotations of elements and relations, ground-truth parses, and more than 15,000 multiple-choice questions.
  • DPG representation: nodes are one of four constituent types (blobs, text boxes, arrows, arrow heads). Edges are one of ten relationship classes, such as inter-object linkage, arrow descriptor and image title.
  • Parsing: Dsdp-Net was trained on about 400,000 sampled relationship sequences. It is scored with the Jaccard Index for Graphs (JIG) and clearly beats the Greedy and A* baselines. A YOLO-style CNN that predicts the whole graph directly gave no meaningful output.
  • Question answering: Dqa-Net uses 300-dimensional GloVe embeddings and a small LSTM with attention over DPG relations. It beats the LSTM Q+I VQA baseline (which uses VGG-19 image features) whether that baseline is trained on VQA or on AI2D.
  • Release: AI2D and the baselines are public. The authors' suggested next step is adding diagrammatic and commonsense knowledge to diagram QA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.07396 · arXiv 1603.07396

@misc{kembhavi2016diagram,
  title = {{A Diagram Is Worth A Dozen Images}},
  author = {Kembhavi, Aniruddha and Salvato, Mike and Kolve, Eric and Seo, Minjoon and Hajishirzi, Hannaneh and Farhadi, Ali},
  year = {2016},
  doi = {10.48550/arxiv.1603.07396},
  eprint = {1603.07396},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.07396}
}

References

48 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Srihari, R.K.: Computational models for integrating linguistic and visual information: A survey. Artif. Intell. Rev. 8 (1994) 349–369 find
  2. Ferguson, R.W., Forbus, K.D.: Telling juxtapositions: Using repetition and alignable difference in diagram understanding. Advances in Analogy Research (1998) find
  3. Watanabe, Y., Nagao, M.: Diagram understanding using integration of layout information and textual information. In: ACL. (1998) find
  4. O’Gorman, L., Kasturi, R.: Document Image Analysis. IEEE Computer Society Executive Briefings (1997) find
  5. Futrelle, R.P., Shao, M., Cieslik, C., Grimes, A.E.: Extraction, layout analysis and classification of diagrams in pdf documents. In: ICDAR. (2003) find
  6. Seo, M.J., Hajishirzi, H., Farhadi, A., Etzioni, O.: Diagram understanding in geometry questions. In: AAAI. (2014) find
  7. von Engelhardt, J.: The language of graphics: A framework for the analysis of syntax and meaning in maps, charts and diagrams. Yuri Engelhardt (2002) find
  8. Zitnick, C.L., Parikh, D.: Bringing semantics into focus using visual abstraction. In: CVPR. (2013) find
  9. Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., Parikh, D.: Yin and yang: Balancing and answering binary visual questions. CoRR abs/1511.05099 (2015) — [arXiv:1511.05099](https://arxiv.org/abs/1511.05099) find
  10. Vedantam, R., Lin, X., Batra, T., Zitnick, C.L., Parikh, D.: Learning common sense through visual abstraction. In: ICCV. (2015) find
  11. Antol, S., Zitnick, C.L., Parikh, D.: Zero-shot learning via visual abstraction. In: ECCV. (2014) find
  12. Tu, Z., Chen, X., Yuille, A.L., Zhu, S.C.: Image parsing: Unifying segmentation, detection, and recognition. In: CLOR. (2003) find
  13. Zhu, S.C., Mumford, D.: A stochastic grammar of images. Foundations and Trends in Computer Graphics and Vision 2 (2006) 259–362 find
  14. Martinovic, A., Gool, L.J.V.: Bayesian grammar learning for inverse procedural modeling. In: CVPR. (2013) find
  15. Pirsiavash, H., Ramanan, D.: Parsing videos of actions with segmental grammars. In: CVPR. (2014) find
  16. Choi, W., Chao, Y.W., Pantofaru, C., Savarese, S.: Understanding indoor scenes using 3d geometric phrases. In: CVPR. (2013) find
  17. Socher, R., Lin, C.C.Y., Ng, A.Y., Manning, C.D.: Parsing natural scenes and natural language with recursive neural networks. In: ICML. (2011) find
  18. Weston, J., Bordes, A., Chopra, S., Mikolov, T.: Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR abs/1502.05698 (2015) find our card
  19. Richardson, M., Burges, C.J.C., Renshaw, E.: Mctest: A challenge dataset for the open-domain machine comprehension of text. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A mee… find our card
  20. Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching machines to read and comprehend. In: Advances in Neural Information Processing Systems. (2015) 1684–1692 find our card
  21. Sukhbaatar, S., Weston, J., Fergus, R., et al.: End-to-end memory networks. In: Advances in Neural Information Processing Systems. (2015) 2431–2439 find
  22. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 2425–2433 find
  23. Ren, M., Kiros, R., Zemel, R.: Exploring models and data for image question answering. In: Advances in Neural Information Processing Systems. (2015) 2935– 2943 find
  24. Zhu, Y., Groth, O., Bernstein, M.S., Fei-Fei, L.: Visual7w: Grounded question answering in images. CoRR abs/1511.03416 (2015) find our card
  25. Andreas, J., Rohrbach, M., Darrell, T., Klein, D.: Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705 (2016) find our card
  26. Noh, H., Seo, P.H., Han, B.: Image question answering using convolutional neural network with dynamic parameter prediction. arXiv preprint arXiv:1511.05756 (2015) — [arXiv:1511.05756](https://arxiv.org/abs/1511.05756) find
  27. Horn, R.: Visual language: Global communication for the 21st century. Century. Brainbridge Island, WA, MacroVU (1998) find
  28. Card, S.K., Mackinlay, J.D., Shneiderman, B.: Readings in information visualization: using vision to think. Morgan Kaufmann (1999) find
  29. Twyman, M.: A schema for the study of graphic language (tutorial paper). In: Processing of visible language. Springer (1979) 117–150 find
  30. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9 (1997) 1735–1780 find
  31. Arbelaez, P.A., Pont-Tuset, J., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping. In: CVPR. (2014) find
  32. Zitnick, C.L., r, P.D.: Edge boxes: Locating object proposals from edges. In: ECCV. (2014) find
  33. Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. 34 (2012) 2189–2202 find
  34. Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. International Journal of Computer Vision 104 (2013) 154–171 find
  35. Kokkinos, I.: Highly accurate boundary detection and grouping. In: CVPR. (2010) find
  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014) find our card
  37. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. (2012) find
  38. Microsoft: Project oxford. https://www.projectoxford.ai/ ↗
  39. de Campos, T.E., Babu, B.R., Varma, M.: Character recognition in natural images. In: Proceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal. (February 2009) find
  40. Tesseract: Open source ocr engine. https://github.com/tesseract-ocr/tesseract ↗
  41. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML. (2010) find
  42. Tieleman, T., Hinton, G.E.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning (2012) find
  43. Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: Unified, real-time object detection. CoRR abs/1506.02640 (2015) — [arXiv:1506.02640](https://arxiv.org/abs/1506.02640) find
  44. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP). (2014) 1532–1543 find
  45. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014) find our card
  46. Chollet, F.: Keras. https://github.com/fchollet/keras (2015) ↗
  47. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Sc… find
  48. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 1… find