Elamur Research⌕
☰
home / papers / 2016 / Image-to-Markup Generation with Coarse-to-Fine Attention

Image-to-Markup Generation with Coarse-to-Fine Attention

Paper · 2016 · Yuntian Deng, Anssi Kanervisto, Jeffrey Ling et al.

Summary

The paper treats recovering LaTeX source from an image of a typeset formula as a form of structured OCR. Unlike plain text recognition, the output need not follow the image left to right, because subscripts, fractions and nesting must be encoded. The authors build a public benchmark, IM2LATEX-100K, of formulas taken from published scientific articles and rendered as images. Their model, IM2TEX, uses a CNN feature grid and runs an LSTM along each row, with trainable per-row initial states that act as positional embeddings. An attention-based LSTM decoder then emits markup tokens. It needs no hand-written grammar of the target language. To make attention cheaper, they add a coarse-to-fine layer: a coarse view first selects a small region, and fine attention runs only inside it. Training uses REINFORCE (hard) or sparsemax. Evaluation renders the predicted LaTeX and checks it against the input image. The system reproduces the exact image in over three quarters of test cases, well ahead of classical math OCR tools. This shows that general attention models can handle layout-heavy recognition.

Key points

  • Dataset and accuracy: On IM2LATEX-100K, standard-attention IM2TEX reaches 77.46% exact image match.
  • Row encoder matters: Removing it drops match to 53.53%, while positional embeddings add only a small gain.
  • Coarse-to-fine savings: Fine-attention lookups fall from 355 to 16 with hard C2F (74.90%) and to 74 with sparsemax (76.15%). Coarse-only attention collapses to 44.40%.
  • Handwriting transfer: After pretraining on synthetic handwriting, the model fine-tunes on CROHME 2013/2014 and exceeds 30% exact match. This is comparable to the top systems except MyScript, which uses private in-domain data.
  • Data needs and errors: The model is data-hungry, needing about 16k training examples to pass 50% accuracy. Its remaining errors are mostly font or bracket-size mismatches.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksComputer Vision and Pattern Recognition · 3,621 worksComputational Theory and Mathematics · 496 works

Cite

Cite as Yuntian Deng, Anssi Kanervisto, Jeffrey Ling and Alexander M. Rush. 2016. Image-to-Markup Generation with Coarse-to-Fine Attention. arXiv:1609.04938.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1609.04938 · arXiv 1609.04938

@misc{deng2016image,
  title = {{Image-to-Markup Generation with Coarse-to-Fine Attention}},
  author = {Yuntian Deng, Anssi Kanervisto, Jeffrey Ling and Alexander M. Rush},
  year = {2016},
  doi = {10.48550/arxiv.1609.04938},
  eprint = {1609.04938},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.04938}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, Robert H. Syntax-directed recognition of hand-printed two-dimensional mathematics. In Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436–459. ACM, 1967. find
  2. Andrychowicz, Marcin and Kurach, Karol. Learning Efficient Algorithms with Hierarchical Attentive Memory. CoRR, abs/1602.0, 2016. find
  3. Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray. Multiple Object Recognition with Visual Attention. Proceedings of the International Conference on Learning Representations (ICLR), 2015. find
  4. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  5. Belaid, Abdelwaheb and Haton, Jean-Paul. A syntactic approach for handwritten mathematical formula recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1):105–111, 1984. find
  6. Bengio, Emmanuel, Bacon, Pierre-Luc, Pineau, Joelle, and Precup, Doina. Conditional Computation in Neural Networks for faster models. CoRR, abs/1511.06297, 2015. URL http://arxiv.org/abs/1511.06297. — [arXiv:1511.06297](https://arxiv.org/abs/1511.06297) ↗
  7. Chan, Kam-Fai and Yeung, Dit-Yan. Mathematical expression recognition: a survey. IJDAR, 3(1):3–15, 2000. doi: 10.1007/PL00013549. URL http://dx.doi.org/10.1007/PL00013549. — [doi:10.1007/pl00013549](https://doi.org/10.1007/pl00013549) ↗
  8. Ciresan, Dan Claudiu, Meier, Ueli, Gambardella, Luca Maria, and Schmidhuber, Jürgen. Deep, big, simple neural nets for handwritten digit recognition. Neural computation, 22(12):3207–3220, 2010. find
  9. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011. find
  10. Gehrke, Johannes, Ginsparg, Paul, and Kleinberg, Jon. Overview of the 2003 kdd cup. ACM SIGKDD Explorations Newsletter, 5(2):149–151, 2003. find
  11. Graves, Alex, Fernández, Santiago, Gomez, Faustino, and Schmidhuber, Jürgen. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369–376. ACM, 2006. find
  12. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  13. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning, pp. 448–456, 2015. find
  14. Jaderberg, Max, Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Deep structured output learning for unconstrained text recognition. ICLR, 2015. find
  15. Jaderberg, Max, Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Reading text in the wild with convolutional neural networks. International Journal of Computer Vision, 116(1):1–20, 2016. find
  16. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137, 2015.
  17. Klein, Guillaume, Kim, Yoon, Deng, Yuntian, Senellart, Jean, and Rush, Alexander M. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810, 2017.
  18. Lee, Chen-Yu and Osindero, Simon. Recursive recurrent nets with attention modeling for ocr in the wild. arXiv preprint arXiv:1603.03101, 2016. find
  19. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. EMNLP, 2015.
  20. Martins, André FT and Astudillo, Ramón Fernandez. From softmax to sparsemax: A sparse model of attention and multi-label classification. CoRR, abs/1602.02068, 2016. find
  21. Mei, Hongyuan, Bansal, Mohit, and Walter, Matthew R. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. Proceedings of NAACL-HLT, pp. 1–11, 2016.
  22. Miller, Erik G and Viola, Paul A. Ambiguity and constraint in mathematical expression recognition. In AAAI/IAAI, pp. 784–791, 1998. find
  23. Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, and koray Kavukcuoglu. Recurrent models of visual attention. Advances in Neural Information Processing Systems, pp. 2204—-2212, 2014. find
  24. Mouchere, Harold, Viard-Gaudin, Christian, Zanibbi, Richard, Garain, Utpal, Kim, Dae Hwan, and Kim, Jin Hyung. Icdar 2013 crohme: Third international competition on recognition of online handwritten mathematical expressions. In 2013 12th International Conference on Document Analysis and Recognition,… find
  25. Mouchere, Harold, Viard-Gaudin, Christian, Zanibbi, Richard, and Garain, Utpal. Icfhr 2014 competition on recognition of on-line handwritten mathematical expressions (crohme 2014). In Frontiers in handwriting recognition (icfhr), 2014 14th international conference on, pp. 791–796. find
  26. Nagabhushan, P and Alaei, Alireza. Tracing and straightening the baseline in handwritten persian/arabic text-line: A new approach based on painting-technique. International Journal on Computer Science and Engineering, 2 (4):907–916, 2010. find
  27. Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu, Wei-Jing. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics, 2002.
  28. Rae, Jack, Hunt, Jonathan J, Danihelka, Ivo, Harley, Timothy, Senior, Andrew W, Wayne, Gregory, Graves, Alex, and Lillicrap, Tim. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes. In Lee, D D, Sugiyama, M, Luxburg, U V, Guyon, I, and Garnett, R (eds.), Advances in Neural Informa… find
  29. Raphael, Christopher. Coarse-to-fine dynamic programming. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(12):1379–1390, 2001. find
  30. Schulman, John, Heess, Nicolas, Weber, Theophane, and Abbeel, Pieter. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pp. 3528–3536, 2015. find
  31. Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc, Hinton, Geoffrey, and Dean, Jeff. Outrageously Large Neural Networks: the Sparsely-Gated Mixture-of-Experts Layer. Proceedings of the International Conference on Learning Representations (ICLR), 2017. find
  32. Shi, Baoguang, Bai, Xiang, and Yao, Cong. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. arXiv preprint arXiv:1507.05717, 2015. — [arXiv:1507.05717](https://arxiv.org/abs/1507.05717) find
  33. Shi, Baoguang, Bai, Xiang, and Yao, Cong. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016. find
  34. Suzuki, Masakazu, Tamari, Fumikazu, Fukuda, Ryoji, Uchida, Seiichi, and Kanahori, Toshihiro. Infty: an integrated ocr system for mathematical documents. In Proceedings of the 2003 ACM symposium on Document engineering, pp. 95–104. find
  35. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156– 3164, 2015.
  36. Wang, Tao, Wu, David J, Coates, Andrew, and Ng, Andrew Y. End-to-end text recognition with convolutional neural networks. In Pattern Recognition (ICPR), 2012 21st International Conference on, pp. 3304–3308. find
  37. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  38. Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun, Courville, Aaron, Salakhudinov, Ruslan, Zemel, Rich, and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of The 32nd International Conference on Machine Learning, pp. 2048–2057, 2015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deng-2016-image-markup-generation-coarse-fine/