Elamur Research⌕
☰
home / papers / 2026 / PatentVision: A multimodal method for drafting patent applications

PatentVision: A multimodal method for drafting patent applications

Paper · 2026 · Ruo Yang, Sai Krishna Reddy Mudhiganti and Manali Sharma

Summary

Writing a patent specification requires turning claims and drawings into a detailed, legally sound technical text. Earlier automatic approaches worked only with text, typically one section at a time, and ignored the figures. The authors present PatentVision, an extension of the text-only PatentFormer pipeline. It feeds a fine-tuned Large Vision-Language Model three inputs: the patent drawing itself, enriched claim text, and an instruction prompt. The enriched claim text includes tags marking component names and numbers, figure numbers, parent-claim context and paragraph positions.

Training data is built by linking each claim feature to a specification paragraph that describes a single figure. The model then learns to produce that paragraph. Gemma 3, LLAVA 1.6 and LLaMA 3.2 were tested as base models, with LoRA fine-tuning. Across many automatic metrics, the multimodal setup beat text-only generation, and Gemma 3 was the strongest backbone. This suggests that diagrams carry useful structural information. It also points to a way of reducing manual drafting effort, although the authors note that human verification remains necessary.

Key points

  • Paragraph alignment: paragraphs are matched to figure descriptions by shared figure numbers, and to claim features by averaging cosine similarity and BLEU. Only paragraphs that reference a figure are kept.
  • Evaluation metrics: BertScore, BLEU, chrF, METEOR, NIST, ROUGE-1/2/L/Lsum and WER.
  • Using raw images: PatentVision outperformed PatentFormer even when PatentFormer was given textual figure descriptions, which indicates the model extracts useful content directly from the images.
  • Ablation results:
    • Mid-range LoRA ranks (32–128) worked best.
    • Rank 256 failed to converge for LLAVA 1.6 and LLaMA 3.2.
    • For Gemma 3, three epochs were optimal and four led to overfitting.
    • Image resolution was tested from 256 to 4096 px, and higher resolution generally helped.
  • Limitations: generation runs per claim, so cost grows linearly with the number of claims. Full conversational editing is planned but not yet available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Samsung Semiconductor.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksBusiness, Management and Accounting · 32 worksManagement of Technology and Innovation · 6 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Ruo Yang, Sai Krishna Reddy Mudhiganti and Manali Sharma. 2026. PatentVision: A multimodal method for drafting patent applications. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.29

@inproceedings{yang2026patentvision,
  title = {{PatentVision: A multimodal method for drafting patent applications}},
  author = {Ruo Yang, Sai Krishna Reddy Mudhiganti and Manali Sharma},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.29},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.29}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dana Aubakirova, Kim Gerdes, and Lufei Liu. 2023. Patfig: Generating short and long captions for patent figures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2843– 2849. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  4. Dimitrios Christofidellis, Antonio Berrios Torres, Ashish Dave, Manuel Roveri, Kristin Schmidt, Sarath Swaminathan, Hans Vandierendonck, Dmitry Zubarev, and Matteo Manica. 2022. Pgt: a prompt based generative transformer for the patent domain. In ICML 2022 Workshop on Knowledge Retrieval and Languag… find
  5. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, page 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Shi Guoliang, Zhou Shu, Wang Yunfeng, Shi Chunjiang, and Liu Liang. 2023. Generating patent text abstracts based on improved multi-head attention mechanism. Data Analysis and Knowledge Discovery, 7(6):61– 72. find
  8. Christopher M Holman. 2024. The us patent and trademark office’s response to recent developments in artificial intelligence. Biotechnology Law Report. find
  9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  10. Lekang Jiang, Caiqi Zhang, Pascal A Scherz, and Stephan Goetz. 2024. Can large language models generate high-quality patent claims? arXiv preprint arXiv:2406.19465. find
  11. LEE Jieh-Sheng. 2022. The effectiveness of bidirectional generative patent language models. In Legal Knowledge and Information Systems: JURIX 2022: The Thirty-fifth Annual Conference, Saarbrücken, Germany, 14-16 December 2022, volume 362, page 194. IOS Press. find
  12. Jieh-Sheng Lee. 2020a. Controlling patent text generation by structural metadata. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3241–3244. find
  13. Jieh-Sheng Lee. 2020b. Measuring and controlling text generation by semantic search. In Companion Proceedings of the Web Conference 2020, pages 269– 273. find
  14. Jieh-Sheng Lee. 2020c. Patent transformer: A framework for personalized patent claim generation. In CEUR Workshop Proceedings, volume 2598. CEUR-WS. find
  15. Jieh-Sheng Lee. 2023. Evaluating generative patent language models. World Patent Information, 72:102173. find
  16. Jieh-Sheng Lee and Jieh Hsiang. 2020a. Patent claim generation by fine-tuning openai gpt-2. World Patent Information, 62:101983. find
  17. Jieh-Sheng Lee and Jieh Hsiang. 2020b. Patenttransformer-1.5: Measuring patent claim 4 generation by span relevancy. In New Frontiers in https://www.federalregister.gov/documents/2024/04/11/2024-Artificial Intelligence, pages 20–33, Cham. Springer 07629/guidance-on-use-of-artificial-intelligence-bas… ↗
  18. Jieh-Sheng Lee and Jieh Hsiang. 2020c. Prior art search and reranking for generated patent text. arXiv preprint arXiv:2009.09132. — [arXiv:2009.09132](https://arxiv.org/abs/2009.09132) find
  19. Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1–39. find
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  21. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507, Geneva, Switzerland. COLING.
  22. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. find
  23. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. pages 311–318.
  25. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  26. Juanyan Wang, Sai Krishna Reddy Mudhiganti, and Manali Sharma. 2024. Patentformer: A novel method to automate the generation of patent applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1361–1380. find
  27. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. find
  28. J.P. Woodard and J.T. Nelson. 1982. An information theoretic measure of speech recognition performance. find
  29. Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021. Hi-transformer: Hierarchical interactive transformer for efficient and effective long document modeling. arXiv preprint arXiv:2106.01040. find
  30. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297.
  31. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  32. Changsheng Zhu, Xin Zheng, and Wenfang Feng. 2023. An automatic generation method of patent specification abstract based on" extraction-abstraction" model. In 2023 IEEE 3rd International Conference on Power, Electronics and Computer Applications (ICPECA), pages 196–200. IEEE. find
  33. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  35. Cinthia M Souza, Magali RG Meireles, and Paulo EM Almeida. 2021. A comparative study of abstractive and extractive summarization techniques to label subgroups on patent dataset. Scientometrics, 126(1):135– 156. find
  36. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-patentvision-multimodal-method-drafting-patent/