Elamur Research⌕
☰
home / papers / 2026 / Constructing a Japanese Claim Decomposition Dataset for Fact-Checking of LLM-Generated Texts

Constructing a Japanese Claim Decomposition Dataset for Fact-Checking of LLM-Generated Texts

Summary

LLM outputs can contain hallucinations. A common way to fact-check them is to first split the text into atomic, self-contained claims, then retrieve evidence for each claim and predict a verdict. How well the text is split strongly affects the final checking accuracy. Earlier resources mostly had annotators review claims that an LLM had already decomposed, and annotators often disagreed. The authors build a manually annotated Japanese claim decomposition dataset, where Japanese serves as a middle-resource contrast to English. Each LLM-generated text is paired with human-written claims, and each claim is labelled check-worthy or not check-worthy. Annotators follow detailed guidelines built on three principles: claims must be understandable without context, the original content must be recoverable from them, and their wording should stay close to the source. Agreement between annotators and the authors' reference decompositions was high. In prompt-based decomposition experiments, combining guidelines with high-quality few-shot examples gave the best results, outperforming FactScore and R-ND prompts. The guidelines are meant to transfer to other languages and to be usable by non-experts.

Key points

  • The dataset has three components: LLM-generated Japanese texts, manually decomposed claims, and check-worthiness labels (check-worthy vs. not check-worthy, after Hassan et al., 2017).
  • The guidelines aim to improve evidence retrieval, reduce ambiguity in verdict prediction, and limit variation between annotators.
  • Annotators were compared with the authors' own decompositions of 17 texts. The F1 was about 0.52 with exact match and about 0.76 with fuzzy match.
  • In the prompting experiments, fuzzy-match F1 on content words rose from 0.374 (zero-shot base) to 0.534 (guideline + 8-shot). With 8 shots, FactScore reached 0.494 and R-ND reached 0.499.
  • Some remaining disagreements come from the guidelines not fully fixing the atomic granularity, so more than one valid split is sometimes possible. The authors state that the dataset and guidelines will be released publicly (LREC 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5nsactrpnuu6

@inproceedings{masano2026constructing,
  title = {{Constructing a Japanese Claim Decomposition Dataset for Fact-Checking of LLM-Generated Texts}},
  author = {Masano, Miwa and Keyaki, Ribeka and Keyaki, Atsushi and Minamoto, Rei and Horio, Kaito and Kiyomaru, Hirokazu and Nakayama, Kouta and Tachibana, Hideyuki and Kawahara, Daisuke},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5nsactrpnuu6},
  url = {https://doi.org/10.63317/5nsactrpnuu6}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop on F… find
  2. Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, and Tal Schuster. 2023. PropSegmEnt: A large-scale corpus for proposition-level segmentation and entailment recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8874–8893, Toronto, Canada. Association for… find
  3. H. Paul Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3, pages 41–58. Academic Press, New York. find
  4. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206. find our card
  5. Martin Haspelmath. 2004. Coordinating constructions: An overview. In Martin Haspelmath, editor, Coordinating Constructions, volume 58 of Typological Studies in Language, pages 3–39. John Benjamins, Amsterdam. find
  6. Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pages 1803–1812, New York, NY, USA… find
  7. Qisheng Hu, Quanyu Long, and Wenya Wang. 2025. Decomposition dilemmas: Does claim decomposition boost or burden fact-checking performance? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  8. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2). find our card
  9. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. 2025. Why language models hallucinate. find
  10. Alkis Kalavasis, Anay Mehrotra, and Grigoris Velegkas. 2025. On the limits of language generation: Trade-offs between hallucination and mode-collapse. In Proceedings of the 57th Annual ACM Symposium on Theory of Computing, STOC ’25, page 1732â��1743, New York, NY, USA. Association for Computing Mach… find
  11. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-world entailment for claims in Wikipedia. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7561– 7583, Singapore. Association for Computational Linguistics. find
  12. Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5430–5443, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  13. Harold William Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97. find
  14. Godehard Link. 1983. The logical analysis of plurals and mass terms: A lattice-theoretical approach. In R. Bäuerle, C. Schwarze, and A. von Stechow, editors, Meaning, Use, and Interpretation of Language, pages 302–323. de Gruyter, Berlin. find
  15. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L… find our card
  16. Kushan Mitra, Dan Zhang, Sajjadur Rahman, and Estevam Hruschka. 2025. FactLens: Benchmarking fine-grained fact verification. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18085–18096, Vienna, Austria. Association for Computational Linguistics. find
  17. Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVeriTeC: A dataset for real-world claim verification with evidence from the web. In Advances in Neural Information Processing Systems, volume 36, pages 65128– 65167. Curran Associates, Inc. find
  18. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  19. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 1– 6, Hong Kong, China. Association for Computational Linguistics. find
  20. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2024. Factcheck-bench: Fine-grained evaluation benchmark for automat… find
  21. Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, and Benjamin Van Durme. 2024. A closer look at claim decomposition. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 153–175, Mexico City, Mexico. Association for Computational Linguistics. find
  22. Etsuyo Yuasa and Jerry M. Sadock. 2002. Pseudo-subordination: A mismatch between syntax and semantics. Journal of Linguistics, 38(1):87–111. find
  23. Xia Zeng, Amani S. Abumansour, and Arkaitz Zubiaga. 2021. Automated fact-checking: A survey. Language and Linguistics Compass, 15(10):e12438. find
  24. Language Resource References Alibaba. 2025. Qwen/Qwen3-32B. distributed via Hugging Face Hub. PID https://huggingface.co/Qwen/Qwen3-32B. ↗
  25. Google. 2025. google/gemma-3-27b-it. distributed via Hugging Face Hub. PID https://huggingface.co/google/gemma-3-27b-it. ↗
  26. LLM-jp. 2024. llm-jp/llm-jp-3-13b-instruct. distributed via Hugging Face Hub. PID https://huggingface.co/llm-jp/llm-jp-3-13b-instruct. ↗
  27. LLM-jp. 2025. LLM-jp Chatbot Arena Conversations Dataset. https://huggingface.co/datasets/llm-jp/llm-jp-chatbot-arena-conversations. ↗
  28. OpenAI. 2024. GPT-4o System Card. OpenAI. PID https://cdn.openai.com/gpt-4o-system-card.pdf. ↗ our card
  29. Jun Suzuki and others. AIO Official Dataset Version 2.0. https://sites.google.com/view/project-aio/home. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/masano-2026-constructing-japanese-claim-decomposition-dataset/