Elamur Research⌕
☰
home / papers / 2026 / Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice

Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice

Paper · 2026 · Qiyao Wang, Guhong Chen, Hongbo Wang et al.

Summary

Most existing resources for evaluating LLMs on intellectual property cover only patents or a few narrow tasks. They rarely test the legal reasoning and decision-making that real IP work requires. To fill this gap, the authors build a task taxonomy for the whole IP field based on Webb's Depth of Knowledge (DOK) theory. The taxonomy is organized into four cognitive levels, with Information Processing as the first, Logical Reasoning as the second, Discriminant Evaluation as the third, and Creative Generation as the fourth. On top of the taxonomy they release IPBench, a bilingual English–Chinese benchmark grounded in US and mainland Chinese law. It includes trademarks, copyrights and trade secrets alongside patents, and its task formats range from classification and multiple-choice questions to open-ended generation. The authors test 19 general-purpose, law-oriented and IP-specific chat and reasoning models in zero-shot, few-shot and chain-of-thought settings. Even the strongest model, DeepSeek-V3, reaches only 75.8%. Closed-source general models outperform open-source domain-specialized ones. The work offers the first broad, cross-jurisdictional testbed for IP-oriented LLMs and shows that current systems remain unreliable for practical IP work.

Key points

  • IPBench has 10,374 data points spanning 20 tasks and 8 IP mechanisms. The data, collected from public sources, and the code are released on GitHub (IPBench/IPBench).
  • Fine-grained patent classification remains very hard. DeepSeek-R1 leads on IPC with 10.8% accuracy, and DeepSeek-V3 leads on CPC with 9.5%.
  • The authors propose LLMScore, an LLM-as-a-judge metric for generation tasks scored on five dimensions. It correlates better with human judgments than BLEU, ROUGE-L or BERTScore (Spearman 0.47 vs. 0.35 for BLEU). Inter-annotator agreement among the human experts was high (Krippendorff's alpha 0.828).
  • More few-shot examples usually raise scores, though Llama3.1-8B is an exception. Chain-of-thought prompting slightly lowers accuracy, by about 0.4–0.6%.
  • A manual review of 300 wrong GPT-4o-mini answers sorted the mistakes into 7 error types. Reasoning errors are the most common (33%), followed by hallucinations (22%) and priority errors (18%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🎓UNSW SydneySydney · Australia34 works · 1,509 citations🔬Shenzhen Institutes of Advanced TechnologyShenzhen · China17 works · 2,599 citations🎓Dalian University of TechnologyDalian · China12 works · 46 citations🎓Shenzhen MSU-BIT UniversityShenzhen · China3 works · 38 citations🎓Shenzhen University of Advanced TechnologyShenzhen · China2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBusiness, Management and Accounting · 32 worksManagement of Technology and Innovation · 6 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin and Min Yang. 2026. Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1000

@inproceedings{wang2026ip,
  title = {{Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice}},
  author = {Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin and Min Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1000},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1000}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zilong Bai, Ruiji Zhang, Linqing Chen, Qijun Cai, Yuan Zhong, Cong Wang, Yan Fang, Jie Fang, Jing Sun, Weikuan Wang, and 1 others. 2024. Patentgpt: A large language model for intellectual property. arXiv preprint arXiv:2404.18255. — [arXiv:2404.18255](https://arxiv.org/abs/2404.18255) find
  2. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang…
  3. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  4. DeepSeek-AI, Daya Guo, Dejian Yang, and Haowei Zhang et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv, abs/2501.12948.
  5. Valentin Knappich, Simon Razniewski, Anna Hätty, and Annemarie Friedrich. 2024. Pap2pat: Towards automated paper-to-patent drafting using chunk-based outline-guided generation. arXiv preprint arXiv:2410.07009. — [arXiv:2410.07009](https://arxiv.org/abs/2410.07009) find
  6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and Bingxuan Wang et al. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437. find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian et al. 2024. The llama 3 herd of models. ArXiv, abs/2407.21783.
  8. Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. 2025. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? Preprint, arXiv:2504.06514. — [arXiv:2504.06514](https://arxiv.org/abs/2504.06514) find
  9. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Informati…
  10. Wanwei He, Jiabao Wen, Lei Zhang, Hao Cheng, Bowen Qin, Yunshui Li, Feng Jiang, Junying Chen, Benyou Wang, and Min Yang. 2023. Hanfei-1.0. https://github.com/siat-nlp/HanFei. ↗
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  12. Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and Min Yang. 2024a. E-EVAL: A comprehensive Chinese k-12 education evaluation benchmark for large language models. In Findings of the Association for Computational Linguist…
  13. Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and 1 others. 2024b. E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models. In ACL (Findings).
  14. OpenAI Aaron Hurst, Adam Lerer, and Adam P. Goucher et al. 2024. Gpt-4o system card. ArXiv, abs/2410.21276.
  15. Fengqing Jiang. 2024. Identifying and mitigating vulnerabilities in llm-integrated applications. Master’s thesis, University of Washington. find
  16. Haitao Li, You Chen, Qingyao Ai, WU Yueyue, Ruizhe Zhang, and LIU Yiqun. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  17. Haitao Li, Jiaying Ye, Yiran Hu, Jia Chen, Qingyao Ai, Yueyue Wu, Junjie Chen, Yifan Chen, Cheng Luo, Quan Zhou, and 1 others. 2025. Casegen: A benchmark for multi-stage legal case documents generation. arXiv preprint arXiv:2502.17943. find
  18. Percy Liang, Rishi Bommasani, and Tony Lee et al. 2022. Holistic evaluation of language models. CoRR, abs/2211.09110. find
  19. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  20. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  21. Shiwen Ni, Minghuan Tan, Yuelin Bai, Fuqiang Niu, Min Yang, Bowen Zhang, Ruifeng Xu, Xiaojun Chen, Chengming Li, and Xiping Hu. 2024. Mozip: A multilingual benchmark to evaluate large language models in intellectual property. In Proceedings of the 2024 Joint International Conference on Computational… find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  23. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  24. Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and Cassidy Hardin et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.00118. find
  25. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. find
  26. Mirac Suzgun, Luke Melas-Kyriazi, Suproteem Sarkar, Scott D Kominers, and Stuart Shieber. 2023. The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications. Advances in neural information processing systems, 36:57908–57946. find
  27. Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, and Simon See. 2025. The curse of cot: On the limitations of chain-of-thought in in-context learning. Preprint, arXiv:2504.05081. — [arXiv:2504.05081](https://arxiv.org/abs/2504.05081) find
  28. M-A-P Team, Xinrun Du, Yifan Yao, and Kaijing Ma et al. 2025. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. Preprint, arXiv:2502.14739. find
  29. You Zuo, Kim Gerdes, Éric Clergerie, and Benoît Sagot. 2024. PatentEval: Understanding errors in patent generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2687–27… find
  30. Qwen Team. 2025b. Qwq-32b: Embracing the power of reinforcement learning. find
  31. Juanyan Wang, Sai Krishna Reddy Mudhiganti, and Manali Sharma. 2024a. Patentformer: A novel method to automate the generation of patent applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1361–1380. find
  32. Qiyao Wang, Jianguo Huang, Shule Lu, Yuan Lin, Kan Xu, Liang Yang, and Hongfei Lin. 2024b. Ipeval: A bilingual intellectual property agency consultation evaluation benchmark for large language models. arXiv preprint arXiv:2406.12386. — [arXiv:2406.12386](https://arxiv.org/abs/2406.12386) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-ip-intelligence-benchmarking-large-language/