Benchmarking Check-Worthiness Models on LLM Generated Claims
Paper · 2026 · Charlie George Roadhouse, Matthew Shardlow and Ashley Williams
Summary
Check-worthiness detectors decide which claims fact-checkers should review first. They are usually trained on human-written statements, so it is unclear how they cope with claims rewritten by LLMs, a growing channel for disinformation. The authors test this on English data from CheckIt (tweets with fine-grained annotations) and CheckThatLab! 2024 (tweets, debates and speeches). Llama 3.1:8B-Instruct paraphrased each claim in three styles while keeping named entities fixed: syntactic restructuring, syntactic complexity and lexical informality.
They benchmarked BERT and RoBERTa fine-tuned on human claims, the zero-shot Llama 3.1:8B, and the commercial ClaimBuster API. They then built "robust" encoders by adding the paraphrases to the training data as adversarial examples. All systems performed worse on synthetic claims, especially complex and informal ones. Adversarial training recovered much of the loss.
SHAP explanations showed that base models depend on shallow cues such as active voice. Robust models focus on factual entities instead. The work argues that fact-checking pipelines need training that accounts for writing style.
Key points
- Four systems were evaluated:
- BERT and RoBERTa, each in a base and a robust variant;
- zero-shot Llama 3.1:8B with greedy decoding;
- ClaimBuster, a BERT-based industry tool.
- Datasets were CheckIt and CheckThatLab! 2024. Encoders were trained on CTL 24 using AdamW (lr 2×10⁻⁵, weight decay 0.01), up to 10 epochs, with early stopping patience 3.
- Paraphrases came from Llama 3.1:8B-Instruct using these sampling settings:
- top-p 0.9;
- temperature 0.8;
- top-k 40;
- repetition penalty 1.1.
- Adversarially trained RoBERTa gained up to +5.22 F1 on CheckIt.
- Code, models and data are released on GitHub (chroadhouse/Benchmarking-Check-Worthiness-Models-on-LLM-Generated-Claims).
Institutions
United KingdomFields, topics and concepts
📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksInformation Systems · 1,730 worksSociology and Political Science · 626 works
Cite
Cite as Charlie George Roadhouse, Matthew Shardlow and Ashley Williams. 2026. Benchmarking Check-Worthiness Models on LLM Generated Claims. Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026.
@inproceedings{roadhouse2026benchmarking,
title = {{Benchmarking Check-Worthiness Models on LLM Generated Claims}},
author = {Charlie George Roadhouse, Matthew Shardlow and Ashley Williams},
booktitle = {Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026},
year = {2026},
doi = {10.63317/5c8w857kxgsn},
url = {https://doi.org/10.63317/5c8w857kxgsn}
}References
- Sajjad Ahmed, Klestia Balla, Knut Hinkelmann, and Flavio Corradini. 2021. Fact Checking: Detection of Check Worthy Statements Through Support Vector Machine and Feed Forward Neural Network. In Advances in Information and Communication, pages 520–535, Cham. Springer International Publishing. find
- Esma Aïmeur, Sabrine Amri, and Gilles Brassard. 2023. Fake news, disinformation and misinformation in social media: a review. Social Network Analysis and Mining, 13(1):30. \#\# Limitations find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/roadhouse-2026-benchmarking-check-worthiness-models-llm/