JustiLM: Few-shot Justification Generation for Explainable Fact-Checking of Real-world Claims
Paper · 2024 · Fengzhu Zeng and Wei Gao
Summary
Prior work on explainable fact-checking mostly produced verdict justifications by summarizing the fact-check article a journalist had already written. This is unrealistic, because no such article exists for a new claim, and it skips evidence search and reasoning. The authors instead generate justifications from evidence retrieved out of a large document collection.
To support this, they build ExClaim from WatClaimCheck. It contains real-world claims with veracity labels, human-written justifications taken from the concluding "Our ruling"/"Our rating" paragraphs, and a searchable pool of reference documents split into 100-word chunks.
Their model, JustiLM, fine-tunes a pre-trained retrieval-augmented generation (RAG) model end-to-end with only a handful of training examples. Fact-check articles supply extra supervision during training but are never seen at inference.
JustiLM beats in-context-learning baselines and the few-shot RAG model Atlas, and it comes close to GPT-4. Its outputs tend to be shorter and more focused than GPT-4's. A simple extension that also predicts veracity improves classification substantially. The work matters because it frames justification generation the way it would actually be deployed.
Key points
- ExClaim has 6,951 real-world claims with labels and justifications. Its retrieval corpus contains 957,949 chunk-level documents pooled across all WatClaimCheck claims.
- The setup is few-shot: K training instances are sampled at random, and no development set is assumed.
- Fact-check articles are used only as auxiliary training signal. At test time, justifications come solely from retrieved evidence.
- JustiLM outperforms Flan-T5, Llama2 (both with ICL) and Atlas, and is competitive with GPT-4. Factual consistency is checked automatically with the NLI-based metric SummaCC.
- Acknowledged limitations:
- There are no passage-level evidence annotations.
- The setup assumes gold evidence is present in the corpus.
- No human evaluation with professional fact-checkers has been done yet.
- Code and data are available at github.com/znhy1024/JustiLM.
Institutions
SingaporeFields, topics and concepts
📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works
Cite
Cite as Fengzhu Zeng and Wei Gao. 2024. JustiLM: Few-shot Justification Generation for Explainable Fact-Checking of Real-world Claims. Transactions of the Association for Computational Linguistics.
@article{zeng2024justilm,
title = {{JustiLM: Few-shot Justification Generation for Explainable Fact-Checking of Real-world Claims}},
author = {Fengzhu Zeng and Wei Gao},
journal = {Transactions of the Association for Computational Linguistics},
year = {2024},
doi = {10.1162/tacl\_a\_00649},
url = {https://doi.org/10.1162/tacl_a_00649}
}References
- Naser Ahmadi, Joohyung Lee, Paolo Papotti, and Mohammed Saeed. 2019. Explainable fact checking with probabilistic answer set programming. In Proceedings of the 2019 Truth and Trust Online Conference, TTO 2019, London, UK. https://doi.org/10.36370/tto .2019.15 — [doi:10.36370/tto](https://doi.org/10.36370/tto) ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2024-justilm-few-shot-justification-generation/