Elamur Research⌕
☰
home / papers / 2025 / TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification

TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification

Paper · 2025 · Junnan Zhu, Min Xiao, Yining Wang et al.

Summary

TROVE frames text provenance as a fine-grained task. For every sentence of a generated or target text, a system must find the exact source sentences it came from, even when the input is long or split across several documents. It must also label how each sentence was derived: quotation, compression, inference, or other (e.g., negation). This differs from citation generation, fact verification and grounded generation, which usually work at document level or with coarser links. The authors built a benchmark from LongBench, LooGLE and CRUD-RAG. It covers 11 scenarios such as QA and summarization, English and Chinese, and inputs from under 5k to over 10k in length. Annotation ran in three steps: candidate retrieval, a first labeling pass by GPT-4o, and human verification. They tested 11 LLMs with direct prompting (DP) and retrieval-augmented (RA) setups. Retrieval helped every model. Bigger models classified relationships more accurately. Closed-source systems scored highest, but open-source models with retrieval came close. Relationship labeling remains hard. The resource supports accountability for generated text in sensitive fields such as law and medicine.

Key points

  • The dataset draws on LongBench, LooGLE and CRUD-RAG and spans 11 scenarios, 2 languages and 3 source-length bands. It is released at github.com/ZNLP/ZNLP-Dataset.
  • One target sentence can map to multiple source sentences, and each link carries its own label. Labels can be mixed, for example compression for two sources and quotation for a third.
  • Gemini-1.5-Pro and GPT-4o achieved the best F1. Retrieval-augmented Qwen2.5-14B sometimes matched them and outperformed Kimi and Gemini-1.5-Pro once sources exceeded 20k.
  • In GPT-4o's errors, quotation was the easiest label and inference the hardest. Under RA, 311 of 672 true inference cases were labeled compression, and only 7 of 28 "other" cases were correct. Quotation→compression errors increased from 401 (DP) to 533 (RA).
  • Limitation: the data comes from existing human-written datasets rather than LLM output, so it contains no hallucination cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

Cite

Cite as Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou and Chengqing Zong. 2025. TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.577

@inproceedings{zhu2025trove,
  title = {{TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification}},
  author = {Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou and Chengqing Zong},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.577},
  url = {https://doi.org/10.18653/v1/2025.acl-long.577}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Aly, Zhiqiang Tang, Samson Tan, and George Karypis. 2024. Learning to generate answers with citations via factual consistency models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 11876– 11896. find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, et al. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
  3. Yushi Bai, Xin Lv, Jiajie Zhang, et al. 2024. Long-Bench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 3119–3137. find
  4. Faeze Brahman, Baolin Peng, Michel Galley, et al. 2022. Grounded keys-to-text generation: Towards factual open-ended generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7397–7413. find
  5. Shuyang Cao and Lu Wang. 2024. Verifiable generation with subsentence-level fine-grained citations. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15584–15596. find
  6. Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2022. Gere: Generative evidence retrieval for fact verification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), page 2184–2189. find
  7. Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024. Complex claim verification with evidence retrieved in the wild. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT),…
  8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, et al. 2023. Vicuna: An open source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  9. Svetlana Churina, Anab Maulana Barik, and Saisamarth Rajesh Phaye. 2024. Improving evidence retrieval on claim verification pipeline through question enrichment. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 64–70. find
  10. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. find
  11. Rosa Falotico and Piero Quatto. 2015. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 49:463–470. find
  12. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6465–6488.
  13. Ameya Godbole, Nicholas Monath, Seungyeon Kim, et al. 2024. Analysis of plan-based retrieval for grounded text generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 13101–13119. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  15. Nianlong Gu and Richard Hahnloser. 2024. Controllable citation sentence generation with language models. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 22–37. find
  16. I-Hung Hsu, Zifeng Wang, Long Le, et al. 2024. CaLM: Contrasting large and small language models to verify grounded generation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12782–12803. find
  17. Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, and Bing Qin. 2024. Learning fine-grained grounded citations for attributed large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pa… find
  18. David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-Farland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics (TACL), 6:391–406.
  19. Anne Lauscher, Brandon Ko, Bailey Kuehl, et al. 2022. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  20. Weitao Li, Junkai Li, Weizhi Ma, and Yang Liu. 2024. Citation-enhanced generation for LLM-based chatbots. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1451–1466. find
  21. Yu Li, Baolin Peng, Yelong Shen, Yi Mao, et al. 2022. Knowledge-grounded dialogue generation with a unified knowledge representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2… find
  22. Hongyuan Lu, Wai Lam, Hong Cheng, and Helen Meng. 2022. On controlling fallback responses for grounded dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2591–2601. find
  23. Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics (TACL), 9:329–345.
  24. Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, et al. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. ACM Transactions on Information Systems, pages 1–32. find
  25. Shrikant Malviya and Stamos Katsigiannis. 2024. SK_DU team: Cross-encoder based evidence retrieval and question generation with improved prompt for the AVeriTeC shared task. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 99–107. lcs. Jurnal Ilmu Komputer dan I… find
  26. Aviv Slobodkin, Eran Hirsch, Arie Cattan, et al. 2024. Attribute first, then generate: Locally-attributable grounded text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 3309–3344. Association for Computational Linguistics.
  27. Kaiqiang Song, Chen Li, Xiaoyang Wang, Dong Yu, and Fei Liu. 2022. Towards abstractive grounded summarization of podcast transcripts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4407–4418. find
  28. Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6181–6190. find
  29. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, et al. 2023. Glm-130b: An open bilingual pre-trained model. Preprint, arXiv:2210.02414.
  30. Liwen Zheng, Chaozhuo Li, Xi Zhang, Yu-Ming Shang, Feiran Huang, and Haoran Jia. 2024. Evidence retrieval is almost all you need for fact verification. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9274–9281. find
  31. Wanjun Zhong, Jingjing Xu, Duyu Tang, et al. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6170–6180.
  32. Anni Zou, Zhuosheng Zhang, and Hai Zhao. 2023. Decker: Double check with heterogeneous knowledge for commonsense fact verification. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11891–11904. find
  33. Biswadip Mandal, Xiangci Li, and Jessica Ouyang. 2024. Contextualizing generated citation texts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3849–3854. find
  34. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, et al. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  35. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find
  36. Munjiah Nur Saadah, Rigga Widar Atmagi, Dyah S Rahayu, and Agus Zainal Arifin. 2013. Information retrieval of text document with weighting tf-idf and find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-trove-challenge-fine-grained-text/