Elamur Research⌕
☰
home / papers / 2026 / Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts

Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts

Paper · 2026 · Miwa Masano, Hirokazu Kiyomaru, Atsushi Keyaki et al.

Summary

Fact-checking pipelines for LLM output usually end with verdict prediction, in which a claim is compared against retrieved evidence. Standard NLI labels handle this poorly because claims and evidence come from unrelated sources. Many pairs look convincingly supported or contradicted without strict logical entailment, and lumping them all into "not enough information" makes a checker less useful. The authors therefore propose six labels: fully-, inferentially-, and partially-supported; fully- and inferentially-refuted; and NEI. The inferential labels cover cases where a reader fills gaps with background knowledge or pragmatic reasoning. Using this scheme, they annotated a Japanese dataset of claim–evidence pairs. The claims come from a prior claim decomposition dataset built on LLM-jp-3 13B Instruct responses. Their study of annotator disagreement points to three causes: unclear limits on acceptable inference, the interpretation of negative cases, and incomplete evidence. In GPT-4o experiments, a chain-of-thought prompt that spells out the verdict procedure gave the best results, raising macro F1 by about 4 points over the baseline. The resource supports Japanese hallucination detection.

Key points

  • The dataset contains 28,147 instances: 13,642 from AIO (AI Official Dataset Version 2.0, QA) and 14,505 from CBA (LLM-jp Chatbot Arena Conversations).
  • Data are split 1:1 into dev and test sets: AIO has 6,818/6,824 instances and CBA has 7,256/7,249.
  • The label set extends Factcheck-Bench-style labels with inferentially-supported and inferentially-refuted; there is deliberately no partial-refutation label.
  • Prompting methods were tested with gpt-4o on the AIO test set, averaged over three runs. CoT outperformed both 6-shot and 12-shot prompting, and adding more examples did not help.
  • Inferentially-refuted was the hardest label for all methods, often confused with fully-refuted or NEI. Labels with high human disagreement were also the ones LLMs predicted worst.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of InformaticsTokyo · Japan37 works · 3,247 citations🎓Waseda UniversityTokyo · Japan20 works · 584 citations🎓Hitotsubashi UniversityTokyo · Japan6 works · 28 citations

Also on the byline: Tokyo University of Technology.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 worksPolitical Science and International Relations · 182 works

Cite

Cite as Miwa Masano, Hirokazu Kiyomaru, Atsushi Keyaki, Kaito Horio, Rei Minamoto, Ribeka Keyaki, Kouta Nakayama, Hideyuki Tachibana and Daisuke Kawahara. 2026. Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.99

@inproceedings{masano2026constructing,
  title = {{Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts}},
  author = {Miwa Masano, Hirokazu Kiyomaru, Atsushi Keyaki, Kaito Horio, Rei Minamoto, Ribeka Keyaki, Kouta Nakayama, Hideyuki Tachibana and Daisuke Kawahara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.99},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.99}
}

References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Proc. of the Fourth Workshop on Fact Ex…
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
  3. Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336. find
  4. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… find
  5. Myroslava Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. SemEval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexical a…
  6. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics (TACL), 10:178–206.
  7. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems,…
  8. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-World Entailment for Claims in Wikipedia. In Proc. of the EMNLP 2023.
  9. Neema Kotonya and Francesca Toni. 2020. Explainable Automated Fact-Checking: A Survey. In Proc. of the COLING 2020. find
  10. Miwa Masano, Ribeka Keyaki, Atsushi Keyaki, Rei Minamoto, Kaito Horio, Hirokazu Kiyomaru, Kouta Nakayama, Hideyuki Tachibana, and Daisuke Kawahara. 2026. Constructing a Japanese Claim Decomposition Dataset for Fact-Checking of LLM-Generated Texts. In International Conference on Language Resources an…
  11. Rodney D. Nielsen, Wayne Ward, James Martin, and Martha Palmer. 2008. Annotating students’ understanding of science concepts. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find
  12. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, and 1 others. 1995. Okapi at TREC-3. British Library Research and Development Department.
  13. Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Proc. of the NeurIPS 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/masano-2026-constructing-japanese-verdict-prediction-dataset/