Elamur Research⌕
☰
home / papers / 2026 / SciCiteVal: A Multi-Domain Dataset for Scientific Citation Verification

SciCiteVal: A Multi-Domain Dataset for Scientific Citation Verification

Summary

Authors in every field sometimes cite sources inaccurately, from honest slips to manipulation by paper mills, and LLMs can invent references outright. Yet there are almost no benchmarks for automatically checking whether a citation faithfully reflects what the cited work says. The authors frame this as an auditing problem, separate from retrieval-style scientific claim verification, and build SciCiteVal to fill the gap.

Each instance pairs a citing sentence with an evidence passage from the cited paper. Labels are Correct, Incorrect (with five error subtypes) or Unrelated. Correct and Incorrect items come from QASA: expert answers serve as citation contexts and their supporting paragraphs as evidence, and annotators deliberately distorted some answers to create the Incorrect cases. Unrelated items are genuine miscitations from earlier work, with the cited abstract used as evidence.

The authors also benchmark several LLMs. However, the performance figures fall in the omitted part of the extracted text and cannot be reported here.

The resource supports citation auditing, explainable verification and fine-grained error classification.

Key points

  • Size and labels: 1,034 citations in total, split into 302 Correct, 302 Incorrect and 430 Unrelated. Incorrect cases are further divided into five fine-grained sub-categories.
  • Sources:
    • Correct and Incorrect items are adapted from QASA (1,798 QA pairs from AI/ML papers), using only its "shallow" and "testing" answerable questions.
    • Unrelated items come from the authors' earlier Cite-screener work, many from the Vickers case documented by da Silva et al. (2023).
  • Construction:
    • Two annotators with NLP backgrounds wrote the distortions and recorded a short rationale for each in a "Twist_reason" field.
    • Original QASA IDs are kept for traceability.
  • File layout: an experiment file with the columns Citation_context, Cited_content, Label and Twist_category, plus a separate, more detailed annotation file. The dataset is public on Hugging Face as birdie0111/SciCiteVal.
  • Domains: social science, biology and machine learning. Unlike SciFact-style datasets, evidence can come from the full text of papers, not just abstracts. The authors present SciCiteVal as complementary to Sarol et al. (2024), whose citation-integrity corpus covers biomedicine only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4m84m2k77g97

@inproceedings{liu2026sciciteval,
  title = {{SciCiteVal: A Multi-Domain Dataset for Scientific Citation Verification}},
  author = {Liu, Qinyue and Zhou, Yongxin and Labbe, Cyril},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4m84m2k77g97},
  url = {https://doi.org/10.63317/4m84m2k77g97}
}

References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Abalkina and Dorothy Bishop. 2023. Paper mills: a novel form of publishing malpractice affecting psychology. Meta-Psychology, 7. find
  2. Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Kalai. 2024. Do language models know when they’re hallucinating references? In Findings of the Association for Computational Linguistics: EACL 2024, pages 912–928, St. Julian’s, Malta. Association for Computational Linguistics. find
  3. AI@Meta. 2024. The llama 3 herd of models. find
  4. Frederique Bordignon. 2022. Critical citations in knowledge construction and citation analysis: from paradox to definition. Scientometrics, 127(2):959–972. find
  5. Guillaume Cabanac, Cyril Labbé, and Alexander Magazinov. 2021. Tortured phrases: A dubious writing style emerging in science. evidence of critical issues affecting established journals. find
  6. Jaime A. Teixeira da Silva, Neil J. Vickers, and Serhii Nazarovets. 2023. From citation metrics to citation ethics: Critical examination of a highly-cited 2017 moth pheromone paper. Scientometrics, 129(1):693–703. find
  7. DeepSeek-AI. 2025. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention. find
  8. Claire Glenton and Benedicte Carlsen. 2019. When “normal” becomes normative: A case study of researchers’ quotation errors when referring to a focus group sample size study. International Journal of Qualitative Methods, 18:1609406919841251. find
  9. Serge Horbach, Kaare Aagaard, and Jesper W. Schneider. 2021. Meta-Research: How problematic citing practices distort science. MetaArXiv aqyhg, Center for Open Science. find
  10. Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. 2025. Sufficient context: A new lens on retrieval augmented generation systems. find
  11. Haixia Liu. 2017. Sentiment analysis of citations using word2vec. CoRR, abs/1704.00177. — [arXiv:1704.00177](https://arxiv.org/abs/1704.00177) find
  12. Jiaying Liu, Xiaomei Bai, Mengying Wang, Suppawong Tuarob, and Feng Xia. 2024a. Anomalous citations detection in academic networks. Artificial Intelligence Review, 57(4). find
  13. Qinyue Liu, Amira Barhoumi, and Cyril Labbé. 2024b. Miscitations in scientific papers: Dataset and detection. pages 53–65. find
  14. Trevor McIntyre and N.S. Haussmann. 2021. Declining citation accuracy in polar research. Polar Record, 57:e43. find
  15. Mistral-AI. 2025. Magistral. find
  16. Josh M. Nicholson, Milo Mordaunt, Patrice Lopez, Ashish Uppala, Domenic Rosati, Neves P. Rodrigues, Peter Grabitz, and Sean C. Rife. 2021. scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies, 2(3):882–898. find
  17. QwenTeam. 2025. Qwen3 technical report. find
  18. Maria Janina Sarol, Shufan Ming, Shruthan Radhakrishna, Jodi Schneider, and Halil Kilicoglu. 2024. Assessing citation integrity in biomedical publications: corpus annotation and nlp models. Bioinformatics, 40(7):btae420. find
  19. Sonita Te, Amira Barhoumi, Martin Lentschat, Frédérique Bordignon, Cyril Labbé, and François Portet. 2022. Citation Context Classification: Critical vs Non-critical. Association for Computational Linguistics, Gyeongju, Republic of Korea. find
  20. Yongxin Zhou, Philippe Mulhem, and Didier Schwab. 2025. Tempperturb-eval: On the joint effects of internal temperature and external perturbations in rag robustness. 7. find our card
  21. Language Resource References Carlos Alvarez, Maxwell Bennett, and Lucy Wang. 2024. Zero-shot scientific claim verification using LLMs and citation text. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 269–276, Bangkok, Thailand. Association for Computational… find
  22. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. find our card
  23. Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10589–10604, Abu Dhabi, United Arab Emirates. As… find
  24. Xanh Ho, Sunisth Kumar, Yun-Ang Wu, Florian Boudin, Atsuhiro Takasu, and Akiko Aizawa. 2025. Table-text alignment: Explaining claim verification against tables in scientific papers. find
  25. Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. 2023. Qasa: Advanced question answering on scientific articles. In Proceedings of the 40th International Conference on Machine Learning. find
  26. Qinyue Liu, Yağmur Öztürk, Tiziri Terkmani, François Portet, and Cyril Labbé. 2025. Cite-screener: A pipeline for citation verification in digital libraries with datasets. pages 158–166. find
  27. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association… find our card
  28. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R. Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/liu-2026-sciciteval-multi-domain-dataset-scientific/