📄 Spark: A System for Scientifically Creative Idea Generation

Summary

Research idea generation is the first step in efforts to automate science, and LLMs have recently shown they can propose novel ideas. This paper presents SPARK, a demonstration system that frames the task in terms of computational creativity (CC) principles. Its pipeline has three modules. XPLOR is a PaperQA-inspired retrieval agent that searches the literature recursively, combining embeddings, FAISS and MMR re-ranking. The Idea Generator pulls concepts and open problems from the retrieved papers and prompts an LLM to draft structured proposals, each with a plan, a title and an abstract. The Filter then has a fine-tuned reviewer model, JUDGE, write several critiques of each idea. A separate LLM reads these critiques and issues an accept or reject verdict along with a utility score. JUDGE was trained on about 600K OpenReview reviews paired with abstracts. The abstracts were rewritten to strip out claims about results, so the model judges the idea itself. The authors report that JUDGE beats its base models. They release the training dataset and outline open CC questions, such as how to judge an idea's utility before any results exist and whether AI can produce transformational ideas.

Key points

• Training data: JUDGE learned from roughly 600K expert reviews harvested from OpenReview. The dataset is public as spiralworks/openreview_wildcard_2025_v2 on Hugging Face.
• Model sizes: JUDGE comes in two versions, wintermute-tiny (based on Qwen-7B) and wintermute-medium (based on Qwen-72B).
  • Both were evaluated on a temporally held-out OpenReview split, using RMSE of review scores among other measures.
  • The paper claims both beat their base models and other contemporary LLMs, but the extracted text gives no numbers.
• Retrieval: XPLOR uses OpenAI text-embedding-3-large, FAISS, LLM relevance scoring on a 1–10 scale, and MMR re-ranking.
  • It loops, refining queries until coverage criteria are met, for example collecting a minimum of five pieces of supporting evidence drawn from more than one source.
  • It can run in interactive or autonomous mode.
• Scale: the pipeline has generated and filtered more than 10,000 AI-related research ideas.
• Limitations: the system has only been tested within AI, and each idea passes through the filter once, with no feedback loop to the generator. Retrieval relies on similarity rather than structured knowledge sources such as ontologies.