Elamur Research⌕
☰
home / papers / 2025 / All That Glitters is Not Novel: Plagiarism in AI Generated Research

All That Glitters is Not Novel: Plagiarism in AI Generated Research

Paper · 2025 · Tarun Gupta and Danish Pruthi

Summary

Recent work, including a large expert study by Si et al. (2024), reports that LLM-based research agents can produce ideas that experts rate as more novel than human ones. Gupta and Pruthi ask whether that apparent novelty hides unattributed reuse of prior work. They asked 13 experts to assume plagiarism and look for source papers. The experts examined 50 LLM-generated documents:

  • 36 new proposals on 12 NLP topics, created with the Si et al. pipeline (Claude 3.5 Sonnet backbone);
  • 4 showcased proposals from Si et al.;
  • 10 papers from "The AI Scientist."

Each document was rated on a 1–5 similarity rubric, and authors of the identified source papers then checked the ratings. About a quarter of the documents turned out to closely mirror or heavily borrow from earlier work, with no credit given. The built-in Semantic Scholar–based filters had not caught these cases. A controlled test with deliberately plagiarized proposals showed that existing automated detectors also fail. The work matters because it challenges claims of machine novelty and points to a growing burden on peer reviewers.

Key points

  • Headline result: 24% of the 50 documents were judged plagiarized.
    • 14% received score 5, a near one-to-one methodological mapping to prior work.
    • 10% received score 4, substantial borrowing from 2–3 papers.
    • Only a small share of the remaining documents looked fully original.
  • Prior exemplars affected: Some previously showcased examples from Si et al. (2024) and Lu et al. (2024a) were plagiarized or very similar to existing work.
  • Author verification: Source-paper authors confirmed the ratings, and in 4 cases they lowered a score by 2 or more points. The final figures use these adjusted scores.
  • Automated detectors fall short: On a synthetic plagiarism dataset, three methods proved inadequate:
    • SSAG (LLM plus Semantic Scholar API retrieval);
    • OpenScholar (45M open-access papers, 237M passage embeddings);
    • Turnitin.
  • Detectable patterns and released data: LLM-generated proposals were less diverse and more formulaic, and simple classifiers might help flag them for review. Code and expert scores are available at github.com/tarun360/AI-Papers-Plagiarism.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Science BangaloreBengaluru · India16 works · 1,462 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Tarun Gupta and Danish Pruthi. 2025. All That Glitters is Not Novel: Plagiarism in AI Generated Research. arXiv:2502.16487.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.16487 · arXiv 2502.16487

@misc{gupta2025all,
  title = {{All That Glitters is Not Novel: Plagiarism in AI Generated Research}},
  author = {Tarun Gupta and Danish Pruthi},
  year = {2025},
  doi = {10.48550/arxiv.2502.16487},
  eprint = {2502.16487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.16487}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing claude 3.5 sonnet. find
  2. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, David Krueger and Roland Memisevic. 2015. Regularizing rnns by stabilizing activations. arXiv preprint arXiv:1511.08400. — [arXiv:1511.08400](https://arxiv.org/abs/1511.08400) find
  3. Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian H… — Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents
  4. Ruochen Li, Liqiang Jing, Chi Han, Jiawei Zhou, and Xinya Du. 2024b. Learning to generate research idea with dynamic control. arXiv preprint arXiv:2412.14626. — [arXiv:2412.14626](https://arxiv.org/abs/2412.14626) find
  5. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024c. Mlr-copilot: Autonomous machine learning research based on large language models agents. arXiv preprint arXiv:2408.14033.
  6. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187. find
  7. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024a. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  8. Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, et al. 2024b. Ai as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against we… — [arXiv:2410.04265](https://arxiv.org/abs/2410.04265) find
  9. Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al. 2024. Large language models surpass human experts in predicting neuroscience results. Nature human behaviour, pages 1–11.
  10. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. Llm4sr: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306.
  11. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex… — GPT-4o System Card
  12. Brian Porter and Edouard Machery. 2024. Ai-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Scientific Reports, 14(1):26133. find
  13. Haosheng Qian, Yixing Fan, Ruqing Zhang, and Jiafeng Guo. 2024. On the capacity of citation generation by large language models. ArXiv, abs/2410.11217. — [arXiv:2410.11217](https://arxiv.org/abs/2410.11217) find
  14. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813.
  15. R Setiawan. 2025. Search icon. Accessed: January 2025. find
  16. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109.
  17. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. 2024. Scicode: A research coding benchmark curated by scientists. arXiv preprint arXiv:2407.13168.
  18. Turnitin, LLC. 2025. Turnitin. Accessed: 2025-02-01. find
  19. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259.
  20. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2024. Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816.
  21. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066.
  22. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2023. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726.
  23. Byeongjun Park, Hyojun Go, Jin-Young Kim, Sangmin Woo, Seokil Ham, and Changick Kim. 2024. Switch diffusion transformer: Synergizing denoising tasks with sparse mixture-of-experts. Preprint, arXiv:2403.09176. — [arXiv:2403.09176](https://arxiv.org/abs/2403.09176) find
  24. Qi Zeng, Mankeerat Sidhu, Hou Pong Chan, Lu Wang, and Heng Ji. 2023. Scientific opinion summarization: Meta-review generation with checklist-guided iterative introspection. arXiv preprint arXiv:2305.14647. — [arXiv:2305.14647](https://arxiv.org/abs/2305.14647) find
  25. L Ray Patterson. 1992. Understanding fair use. Law and contemporary problems, 55(2):249–266. find
  26. Karl Popper. 2013. The poverty of historicism. Routledge. find
  27. Karl Popper. 2014. Conjectures and refutations: The growth of scientific knowledge. routledge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2025-all-that-glitters-novel/