Elamur Research⌕
☰
home / papers / 2023 / AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ

AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ

Paper · 2023 · Jonas Belouadi, Anne Lauscher and Steffen Eger

Summary

Text-to-image systems mostly output raster images, which poorly suit scientific figures that need exact geometry and readable text. Generating vector formats such as SVG directly is also hard because they rely on low-level primitives. The authors propose TikZ as an intermediate target instead. TikZ is a high-level graphics language that compiles to vector graphics, so figure generation becomes a conditional language-modeling task.

To support this, they built DaTikZ, the first large-scale collection of caption-aligned TikZ drawings, with roughly 120k examples. They fine-tuned LLaMA on it and also introduced CLiMA, which adds multimodal CLIP embeddings to LLaMA. These embeddings let the model interpret captions visually and optionally accept images as input.

In human and automatic evaluations, both fine-tuned models produced figures closer to human-made ones than GPT-4 and Claude 2, and CLiMA further improved text-image alignment. The proprietary models tended to output simpler figures and sometimes copied the caption into the image. The framework, AutomaTikZ, is released with model weights and data. It points toward practical tools for drafting scientific figures and teaching TikZ.

Key points

  • DaTikZ contains 119,789 TikZ–caption pairs, all compiling with TeX Live 2023 and dated November 2006 to June 2023. The sources are:
    • arXiv papers: 85,656
    • TeX Stack Exchange: 29,238
    • artificial examples: 3,914
    • curated websites: 981
  • About 62.7% of DaTikZ is augmented data.
  • Captions for TeX Stack Exchange items were written by WizardLM, using each question's title and body as context.
  • In the n-gram novelty analysis, more than 80% of model n-grams were absent from the training data for n > 8, which indicates little memorization. For unigrams, CLiMA 13b and LLaMA 13b copied about 6.5% of caption n-grams, while GPT-4 and Claude 2 copied over 10%.
  • Because GPT-4 and Claude 2 copy caption text into images (typographic attacks), the authors consider image-based CLIPScore the more robust measure for text-rich outputs.
  • The public release omits arXiv drawings under non-redistributable licenses, but the dataset-building code is published so others can recreate the full set. Over 120k arXiv TikZ figures were dropped because they failed to compile.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MannheimMannheim · Germany29 works · 8,347 citations🎓Universität HamburgHamburg · Germany81 works · 5,549 citations🎓Bielefeld UniversityBielefeld · Germany35 works · 1,196 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Jonas Belouadi, Anne Lauscher and Steffen Eger. 2023. AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ. arXiv:2310.00367 · The Twelfth International Conference on Learning Representations, 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.00367 · arXiv 2310.00367

@misc{belouadi2023automatikz,
  title = {{AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ}},
  author = {Jonas Belouadi, Anne Lauscher and Steffen Eger},
  year = {2023},
  doi = {10.48550/arxiv.2310.00367},
  eprint = {2310.00367},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00367}
}

References

135 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. Can we trust the evaluation on ChatGPT? In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 47–54, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.t…
  2. Anthropic. Model card and evaluations for Claude models, 2023. URL https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf. ↗
  3. Haruka Aoki and Kiyoharu Aizawa. SVG vector font generation for chinese characters with transformer. In 2022 IEEE International Conference on Image Processing, ICIP 2022, Bordeaux, France, 16-19 October 2022, pp. 646–650. find
  4. IEEE, 2022. doi: 10.1109/ICIP46576.2022.9897633. URL https://doi.org/10.1109/ICIP46576.2022.9897633. — [doi:10.1109/icip46576.2022.9897633](https://doi.org/10.1109/icip46576.2022.9897633) ↗
  5. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022. 10https://creativecommons.org/licenses 11https://www.gnu.org/licenses/fdl-1.3.en.html 12https://opensource.org/license/mit…
  6. Jonas Belouadi and Steffen Eger. ByGPT5: End-to-end style-conditioned poetry generation with token-free language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7364–7381, Toronto, Canada, July 2023a. Association for Com… find
  7. Jonas Belouadi and Steffen Eger. UScore: An effective approach to fully unsupervised evaluation metrics for machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 358–374, Dubrovnik, Croatia, May 2023b. Association for… find
  8. Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. find
  9. OpenReview.net, 2018. URL https://openreview.net/forum?id=r1lUOzWCW. ↗
  10. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  11. OpenReview.net, 2019. URL https://openreview.net/ forum?id=B1xsqj09Fm. ↗
  12. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023.
  13. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pp. 535–541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: 10.1145/11… — [doi:10.1145/1150402.1150464](https://doi.org/10.1145/1150402.1150464) find
  14. Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lee. Leveraging large language models for scalable vector graphics-driven image understanding, 2023. find
  15. Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. DeepSVG: A hierarchical generative network for vector graphics animation. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems… find
  16. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  17. OpenReview.net, 2023. URL https://openreview.net/pdf?id=TatRHT_1cK. ↗
  18. Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyungh… find
  19. PMLR, 2023. URL https://proceedings.mlr.press/v202/chang23b.html. ↗
  20. Charles Chen, Ruiyi Zhang, Sungchul Kim, Scott Cohen, Tong Yu, Ryan A. Rossi, and Razvan C. Bunescu. Neural caption generation over figures. In Robert Harle, Katayoun Farrahi, and Nicholas D. Lane (eds.), Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computin… find
  21. ACM, 2019a. doi: 10.1145/3341162.3345601. URL https://doi.org/10.1145/3341162.3345601. — [doi:10.1145/3341162.3345601](https://doi.org/10.1145/3341162.3345601) ↗
  22. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Tong Yu, Ryan Rossi, and Razvan Bunescu. Figure captioning with reasoning and sequence-level training, 2019b. find
  23. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan A. Rossi. Figure captioning with relation maps for reasoning. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pp. 1526–1534. find
  24. IEEE, 2020. doi: 10.1109/WACV45572.2020.9093592. URL https://doi.org/10.1109/ WACV45572.2020.9093592. — [doi:10.1109/wacv45572.2020.9093592](https://doi.org/10.1109/wacv45572.2020.9093592) find
  25. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  26. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio…
  27. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning, 2023.
  28. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized llms, 2023.
  29. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering text-to-image generation via transformers. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.)…
  30. Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and better text-to-image generation via hierarchical transformers. In NeurIPS, 2022. URL http://papers.nips.cc/paper_ files/paper/2022/hash/6baec7c4ba0a8734ccbd528a8090cb1f-Abstract-Conference. html. ↗
  31. Aryaz Eghbali and Michael Pradel. CrystalBLEU: Precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394758. doi: 1… — [doi:10.1145/3551349.3556903](https://doi.org/10.1145/3551349.3556903) find
  32. Sebastian Thore Erdweg and Klaus Ostermann. Featherweight TeX and parser correctness. In Brian A. Malloy, Steffen Staab, and Mark van den Brand (eds.), Software Language Engineering - Third International Conference, SLE 2010, Eindhoven, The Netherlands, October 12-13, 2010, Revised Selected Papers,… find
  33. Springer, 2010. doi: 10.1007/978-3-642-19440-5\\_26. URL https://doi.org/10.1007/978-3-642-19440-5_ 26. — [doi:10.1007/978-3-642-19440-5\\_26](https://doi.org/10.1007/978-3-642-19440-5_26) ↗
  34. Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 12873–12883. find
  35. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021. 01268. URL https://openaccess.thecvf.com/content/CVPR2021/html/Esser_Taming_ Transformers_for_High-Resolution_Image_Synthesis_CVPR_2021_paper.html. — [doi:10.1109/cvpr46437.2021](https://doi.org/10.1109/cvpr46437.2021) ↗
  36. Kevin Frans, Lisa B. Soros, and Olaf Witkowski. CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders. In NeurIPS, 2022. URL http://papers.nips.cc/paper_ files/paper/2022/hash/21f76686538a5f06dc431efea5f475f5-Abstract-Conference. html. ↗
  37. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474, 2021. doi: 10.1162/ tacl_a_00437…
  38. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May… find
  39. OpenReview.net, 2023. URL https://openreview.net/pdf? id=hQwb-lbM6EL. ↗
  40. Andrew Gelman, Cristian Pasarica, and Rahul Dodhia. Let’s practice what we preach: Turning tables into graphs. The American Statistician, 56(2):121–130, 2002. doi: 10.1198/000313002317572790. URL https://doi.org/10.1198/000313002317572790. — [doi:10.1198/000313002317572790](https://doi.org/10.1198/000313002317572790) ↗
  41. Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation, 2023. find
  42. Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5356–5364. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR. 2019.00550. URL http://… find
  43. David Ha and Douglas Eck. A neural representation of sketch drawings. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. find
  44. OpenReview.net, 2018. URL https://openreview.net/ forum?id=Hy6GHpkCW. ↗
  45. James Hartley. Single authors are not alone: Colleagues often help. Journal of Scholarly Publishing, 34 (2):108–113, 2003. doi: 10.3138/jsp.34.2.108. URL https://doi.org/10.3138/jsp.34.2.108. — [doi:10.3138/jsp.34.2.108](https://doi.org/10.3138/jsp.34.2.108) ↗
  46. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514–7528, Online and Punta Cana, Dominican Republic, November…
  47. John D. Hobby. MetaPost, 2014. URL https://tug.org/metapost. ↗
  48. Zhi Hong, Aswathy Ajith, James Pauloski, Eamon Duede, Kyle Chard, and Ian Foster. The diminishing returns of masked language models to science. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1270–1283, Toronto, Canada, July 2023. Association for Computational Linguistics… find
  49. Sameera Horawalavithana, Sai Munikoti, Ian Stewart, and Henry Kvinge. SCITUNE: Aligning large language models with scientific multimodal instructions, 2023. find
  50. Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3258–3264, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.… find
  51. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
  52. OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  53. Chieh-Yang Huang, Ting-Yao Hsu, Ryan A. Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, C. Lee Giles, and Ting-Hao (Kenneth) Huang. Summaries as captions: Generating figure captions for scientific documents with automated text summarization. In C. Maria Keet, Hung-Yi Lee, and Sin… find
  54. Ajay Jain, Amber Xie, and Pieter Abbeel. VectorFusion: Text-to-SVG by abstracting pixel-based diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1911–1920. find
  55. IEEE, 2023. doi: 10.1109/CVPR52729.2023.00190. URL https://doi.org/10.1109/CVPR52729.2023.00190. — [doi:10.1109/cvpr52729.2023.00190](https://doi.org/10.1109/cvpr52729.2023.00190) ↗
  56. Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5648– 5656. Computer Vision Foundation / IEE… — [doi:10.1109/cvpr](https://doi.org/10.1109/cvpr) find
  57. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Pr…
  58. OpenReview.net, 2018. URL https://openreview.net/ forum?id=H1mz0OyDz. ↗
  59. Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 10124–10134. find
  60. IEEE, 2023. doi: 10.1109/CVPR52729.2023.00976. URL https://doi.org/10.1109/CVPR52729.2023.00976. ↗
  61. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
  62. Svetlana Kiritchenko and Saif Mohammad. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 465–470, Vancouver, Canada, July 2017.…
  63. Svetlana Kiritchenko and Saif M. Mohammad. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 811–817, S…
  64. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceeding… find
  65. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Ev…
  66. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conferenc… find
  67. AAAI Press, 2020. doi: 10.1609/AAAI.V34I07.6795. URL https://doi.org/10.1609/aaai.v34i07.6795. ↗
  68. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Mach…
  69. PMLR, 2022a. URL https://proceedings.mlr.press/v162/li22n.html. ↗
  70. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas… find
  71. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Ré mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James… — Competition-Level Code Generation with AlphaCode
  72. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014 - 13th European Conferen…
  73. Springer, 2014. doi: 10.1007/978-3-319-10602-1\\_48. URL https://doi.org/10.1007/978-3-319-10602-1_48. — [doi:10.1007/978-3-319-10602-1\\_48](https://doi.org/10.1007/978-3-319-10602-1_48) ↗
  74. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=w0H2xGHlkw. ↗
  75. Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. A learned representation for scalable vector graphics. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp. 7929–7938. find
  76. IEEE, 2019. doi: 10.1109/ICCV.2019.00802. URL https://doi.org/10.1109/ICCV.2019.00802. — [doi:10.1109/iccv.2019.00802](https://doi.org/10.1109/iccv.2019.00802) ↗
  77. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  78. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  79. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press, 2015. doi: 10.1017/CBO9781107337855. — [doi:10.1017/cbo9781107337855](https://doi.org/10.1017/cbo9781107337855) find
  80. Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. Towards layer-wise image vectorization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16293–16302. find
  81. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01583. URL https://doi.org/10.1109/CVPR52688. 2022.01583. — [doi:10.1109/cvpr52688.2022.01583](https://doi.org/10.1109/cvpr52688.2022.01583) ↗
  82. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. Transactions of the Association for Computational Linguistics, 11:652–670, 2023. doi: 10.1162/tacl…
  83. Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A three sample hypothesis test for evaluating generative models. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Mach… find
  84. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, Ap… find
  85. OpenReview.net, 2018. URL https://openreview.net/forum?id=r1gs9JgRZ. ↗
  86. OpenAI. GPT-4 technical report, 2023.
  87. Bryan K. Orme. MaxDiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Software Research Paper Series, 2009. URL https://content.sawtoothsoftware.com/assets/ 8e69929d-a089-4b93-a9f3-d4c64d156642. ↗
  88. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311–318.
  89. ACL, 2002. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040/. ↗
  90. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. find
  91. OpenReview.net, 2022. URL https://openreview.net/forum?id=KmtVD97J43e. ↗
  92. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proc…
  93. PMLR, 2021. URL http://proceedings.mlr.press/v139/radford21a.html. ↗
  94. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Even…
  95. PMLR, 2021. URL http://proceedings.mlr.press/v139/ramesh21a.html. ↗
  96. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents, 2022.
  97. Vikas Raunak and Arul Menezes. Finding memo: Extractive memorization in constrained sequence generation tasks. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5153–5162, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https… find
  98. Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY,… find
  99. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/reed16.html. ↗
  100. Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. Searching for COMETINHO: The little metric that could. In Helena Moniz, Lieve Macken, Andrew Rufener, Loïc Barrault, Marta R. Costa-jussà, Christophe Declercq, Maarit Koponen, Ellie Kemp… find
  101. Juan A. Rodriguez, David Vázquez, Issam H. Laradji, Marco Pedersoli, and Pau Rodríguez. FigGen: Text to scientific figure generation. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023. find
  102. OpenReview.net, 2023a. URL https://openreview.net/pdf?id=Hx_iTXnCR5. ↗
  103. Juan A. Rodriguez, David Vázquez, Issam H. Laradji, Marco Pedersoli, and Pau Rodríguez. OCR-VQGAN: Taming text-within-image generation. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, pp. 3678–3687. find
  104. IEEE, 2023b. doi: 10.1109/WACV56688.2023.00368. URL https://doi.org/10.1109/WACV56688. 2023.00368. — [doi:10.1109/wacv56688.2023.00368](https://doi.org/10.1109/wacv56688.2023.00368) ↗
  105. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674–10685.
  106. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01042. URL https://doi.org/10. 1109/CVPR52688.2022.01042. — [doi:10.1109/cvpr52688.2022.01042](https://doi.org/10.1109/cvpr52688.2022.01042) find
  107. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — Code Llama: Open Foundation Models for Code
  108. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understandin… — Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  109. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9895–9901, Online and Punta Cana, Dominican Republic, No… find
  110. Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Kumar Divvala, and Ali Farhadi. FigureSeer: Parsing result-figures in research papers. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,… find
  111. Springer, 2016. doi: 10.1007/978-3-319-46478-7\\_41. URL https://doi.org/10.1007/978-3-319-46478-7_41. — [doi:10.1007/978-3-319-46478-7\\_41](https://doi.org/10.1007/978-3-319-46478-7_41) ↗
  112. Ashish Singh, Prateek Agarwal, Zixuan Huang, Arpita Singh, Tong Yu, Sungchul Kim, Victor Bursztyn, Nikos Vlassis, and Ryan A. Rossi. FigCaps-HF: A figure-to-caption generative framework and benchmark with human feedback, 2023. find
  113. Peter Stanchev, Weiyue Wang, and Hermann Ney. EED: Extended edit distance measure for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 514–520, Florence, Italy, August 2019. Association for Computational Linguistics. doi:… find
  114. Till Tantau. The TikZ and PGF Packages, 2023. URL https://github.com/pgf-tikz/pgf. ↗
  115. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023a.
  116. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  117. Edward Rolf Tufte. The visual display of quantitative information. Graphics Press, 1992. ISBN 978-0-9613921-0-9. find
  118. Timothy van Zandt. PSTricks: PostScript macros for Generic TeX, 2007. URL https://tug.org/ PSTricks. ↗
  119. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Ga…
  120. PMLR, 2022a. URL https://proceedings.mlr.press/v162/wang22al.html. ↗
  121. Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. CLIP-GEN: Language-free training of a text-to-image generator with CLIP, 2022b. find
  122. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Inform…
  123. Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. IconShop: Text-guided vector icon synthesis with autoregressive transformers, 2023. find
  124. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large language models to follow complex instructions, 2023. find
  125. Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Swarat Chaudhuri and Charles Sutton (eds.), MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022, pp. 1–10.
  126. ACM, 2022. doi: 10.1145/3520312.3534862. URL https://doi.org/10.1145/3520312.3534862. — [doi:10.1145/3520312.3534862](https://doi.org/10.1145/3520312.3534862) ↗
  127. Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. SciCap+: Knowledge augmented dataset to study the challenges of scientific figure captioning, 2023. find
  128. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG-DocOwl: Modularized multimodal large language model for document understanding, 2023. find
  129. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models, 2023.
  130. Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet NL2Code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7443–7464, Toronto, Canada,… find
  131. Han Zhang, Tao Xu, and Hongsheng Li. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 5908–5916. find
  132. IEEE Computer Society, 2017. doi: 10.1109/ICCV.2017.629. URL https://doi.org/10.1109/ICCV.2017.629. — [doi:10.1109/iccv.2017.629](https://doi.org/10.1109/iccv.2017.629) ↗
  133. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023a. find
  134. Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with GPT-4, 2023b. find
  135. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding, 2023c.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/belouadi-2023-automatikz-text-guided-synthesis-scientific/