Elamur Research⌕
☰
home / papers / 2025 / SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings

SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings

Paper · 2025 · Farhana Keya, Gollam Rabby, Prasenjit Mitra et al.

Summary

SCI-IDEA tackles a hard problem for LLM-based research assistants: proposing scientific ideas that are novel, grounded in context, and actually workable. Starting from a researcher's identifier (such as an ORCID ID) or a free-text query, the framework pulls related publications from CORE, arXiv and Semantic Scholar. It then uses LLMs to extract structured facets (objectives, methodologies, evaluation, future work) and infers research gaps from them. Candidate ideas come from zero-shot, zero-shot chain-of-thought or few-shot prompting. Each idea gets two scores: novelty, based on embedding cosine dissimilarity, and surprise, based on negative log-likelihood under a pretrained language model. Ideas that clear both thresholds are flagged as "Aha moments" and refined further with researcher feedback. The authors compare token-level and sentence-level embeddings across GPT-4o, GPT-4.5, DeepSeek-32B and DeepSeek-70B. LLM judges and human experts rate ideas on novelty, excitement, feasibility and effectiveness. The best configurations average roughly 6.8–6.9 out of 10. The work matters as a structured, human-in-the-loop pipeline for ideation, although human ratings show that LLM judges can be overly generous.

Key points

  • Pipeline: facet extraction, research-gap detection, prompted idea generation, then embedding-based "Aha Moment" filtering. The paper's example uses a novelty threshold θn = 0.7 and a surprise threshold θs = 2.0.
  • Token-level embeddings: the best per-model averages are 6.84 (GPT-4o, 2-shot), 6.86 (GPT-4.5, 2-shot), 6.89 (DeepSeek-32B, 2-shot) and 6.84 (DeepSeek-70B, 3-shot).
  • Sentence-level embeddings: the best per-model averages are 6.87 (GPT-4o, 5-shot), 6.86 (GPT-4.5, 3-shot), 6.83 (DeepSeek-32B, zero-shot CoT) and 6.87 (DeepSeek-70B, 5-shot).
  • Prompting and model size: few-shot prompting usually beats zero-shot variants, especially on feasibility and effectiveness. The smaller DeepSeek-32B is competitive overall but less novel: 7.00 under zero-shot versus 7.61 for GPT-4o with 5-shot.
  • Humans vs. LLM judges: human scores can differ sharply from LLM scores. For GPT-4o with 2-shot prompting, the LLM judge gave 6.84 while human experts gave 4.25.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Farhana Keya, Gollam Rabby, Prasenjit Mitra, Sahar Vahdati, Soren Auer and Yaser Jaradeh. 2025. SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings. arXiv:2503.19257.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.19257 · arXiv 2503.19257

@misc{keya2025sci,
  title = {{SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings}},
  author = {Farhana Keya, Gollam Rabby, Prasenjit Mitra, Sahar Vahdati, Soren Auer and Yaser Jaradeh},
  year = {2025},
  doi = {10.48550/arxiv.2503.19257},
  eprint = {2503.19257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.19257}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2. Baek, J., Jauhar, S.K., Cucerzan, S., Hwang, S.J.: Researchagent: Iterative research idea gene…
  2. The Wharton School Research Paper Forthcoming (2023) 9. Gottweis, J., Weng, W.H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., et al.: Towards an ai co-scientist. arXiv preprint arXiv:2502.18864 (2025) 10. Gu, X., Krenn, M.: Interesting scient… — Accelerating scientific discovery with Co-Scientist
  3. SCI-IDEA 12. Guo, S., Shariatmadari, A.H., Xiong, G., Huang, A., Xie, E., Bekiranov, S., Zhang, A.: Ideabench: Benchmarking large language models for research idea generation. arXiv preprint arXiv:2411.02429 (2024) 13. Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models ar…
  4. Springer Science & Business Media (2013) 15. Li, R., Jing, L., Han, C., Zhou, J., Du, X.: Learning to generate research idea with dynamic control. arXiv preprint arXiv:2412.14626 (2024) 16. Li, S., Padilla, S., Bras, P.L., Dong, J., Chantler, M.: A review of llm-assisted ideation. arXiv preprint arX… find
  5. OpenAI Blog (2019), — Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks https://openai.com/blog/ language-unsupervised 26. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. on Natural Language Processing, EMNLP-IJCNLP. ↗
  6. ACL (2019). https://doi.org/10.18653/V1/D19-1410 27. Si, C., Yang, D., Hashimoto, T.: Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109 (2024) 28. Srinivasan, G., Lee, C., Sengupta, A., Panda, P., Sarwar, S.S., Roy, K.: Trai… ↗
  7. F. Keya et al. 29. Uhlmann, E.L., Ebersole, C.R., Chartier, C.R., Errington, T.M., Kidwell, M.C., Lai, C.K., McCarthy, R.J., Riegelman, A., Silberzahn, R., Nosek, B.A.: Scientific utopia iii: Crowdsourcing science. Perspectives on Psychological Science 14(5), 711–733 (2019) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keya-2025-sci-idea-context-aware-scientific/