Elamur Research⌕
☰
home / papers / 2026 / DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification

DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification

Paper · 2026 · Sagnik Sinha and Shreyas Shrestha

Summary

This paper addresses CLEF 2026 CheckThat! Task 2. Each instance has a claim involving numbers or time, its evidence, and several LLM reasoning traces sampled at different temperatures. A system must rank the traces by usefulness and then give a verdict: True, False or Conflicting. The authors cover English (QuanTemp-derived) and Arabic. Both of their systems use the same weak label: a trace counts as positive when its verdict agrees with the gold claim label.

  • Approach I fine-tunes an LLM verifier with LoRA as a binary trace scorer and chooses the verdict by Best-of-N. As an option, it first splits each claim into at most two sub-claims using Llama-3.2-3B-Instruct.
  • Approach II is a small TF-IDF reward model with word and character features plus numeric and temporal overlap and mismatch signals. Its scores are pooled per verdict group.

On English, Approach I has the best overall score, decomposition lowers results, and Approach II does best on the hard Conflicting class. On Arabic, AraBERT beats BERT. The work matters because it rates the reasoning behind a verdict, not only the label.

Key points

  • English test, Approach I: score 41.88, macro-F1 58.98, Recall@5 24.77. With sub-claims these fall to 39.68, 55.79 and 23.57. Approach II reaches 40.50, 58.78 and 22.21.
  • Conflicting F1 (English): Approach II gets 18.02, against 12.74 for Approach I and 13.33 with sub-claims. Conflicting is the hardest class in every setting.
  • Arabic test: AraBERT (aubmindlab/bert-base-arabertv02) gets score 59.13, macro-F1 84.85 and Recall@5 33.41. BERT gets 55.93, 82.32 and 29.54. The Arabic test set has no Conflicting claims.
  • Official metric: the final score is the mean of macro-F1 and Recall@5 (MRR@5 is also tracked). The task also offers 2,808 Spanish claims, which this team did not tackle; the Arabic portion has 3,260 claims.
  • Arabic training setup: LoRA with r=8, α=16 and dropout 0.1 on a frozen encoder. The maximum input length of 512 tokens matches the 95th percentile of training-set lengths. The checkpoint was chosen by validation accuracy. Focal loss was tried and dropped because it hurt results.

Interesting ideas

  • I1. Each trace gets a label for free by checking whether its final verdict matches gold, with about two agreeing and several disagreeing traces sampled per claim; this turns candidate ranking into balanced binary classification without manual annotation.
  • I2. Support is summed across all candidates that back the same verdict instead of trusting only the top-ranked one, and this pooling helped most when evidence was spread across competing answers.
  • I3. Explicit numeric and temporal overlap and mismatch features help a lightweight model notice when quantities or dates in a trace disagree with the claim.
  • I4. Splitting claims into sub-claims with an LLM added noise and lowered scores, which shows that a plausible preprocessing step must be ablated before it is trusted.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): score candidate parses or frame sets from witness parsers and LLM teacher tiers by agreement with gold. These labels could train a ranker that picks the best candidate per sentence.
  • I2 (answer): when several reasoning paths over the text world yield answers, pool their support per answer. A flat, split distribution can be reported explicitly as an ambiguous or conflicting case.
  • I3 (world): compare quantities, units and dates in a question against the stored measures and event order. Mismatches would be explainable failure signals rather than hidden scores.
  • I4 (frames): splitting complex sentences into sub-frames should be measured against whole-sentence frames. It should be kept only where it measurably helps downstream answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States69 works · 22,788 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Information Systems · 523 works Sociology and Political Science · 221 works

🧭 Topics 1 Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Sagnik Sinha and Shreyas Shrestha. 2026. DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification. arXiv:2607.25069.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.25069 · arXiv 2607.25069

@misc{sinha2026ds,
  title = {{DS@GT ARC at CheckThat! 2026: LLM-Based Trace Ranking and Grouped Reward Modeling for Multilingual Numerical Claim Verification}},
  author = {Sagnik Sinha and Shreyas Shrestha},
  year = {2026},
  doi = {10.48550/arxiv.2607.25069},
  eprint = {2607.25069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.25069}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Aïmeur, S. Amri, G. Brassard, Fake news, disinformation and misinformation in social media: a review, Social Network Analysis and Mining 13 (2023) 30. find
  2. Pew Research Center, Social media and news fact sheet, https://www.pewresearch.org/journalism/ fact-sheet/social-media-and-news-fact-sheet/, 2023. Accessed: 2025-05-27. ↗
  3. S. Shaar, N. Georgiev, F. Alam, G. Da San Martino, A. Mohamed, P. Nakov, Assisting the human fact-checkers: Detecting all previously fact-checked claims in a document, arXiv preprint arXiv:2109.07410 (2021). ↗
  4. N. Sagara, Consumer Understanding and Use of Numeric Information in Product Claims, Ph.D. thesis, University of Oregon, Eugene, OR, 2009. find
  5. A. Runewicz, P. M. Ranly, I. Vogel, M. Steinebach, Fraunhofer SIT at CheckThat! 2025: Multi-instance evidence pooling for numerical claim verification, in: Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, 2025. find
  6. M. Heil, A. Pramov, DS@GT at CheckThat! 2025: Evaluating context and tokenization strategies for numerical fact verification, in: Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, 2025. find
  7. P. Nakov, A. Barrón-Cedeño, T. Elsayed, R. Suwaileh, L. Màrquez, W. Zaghouani, P. Atanasova, S. Kyuchukov, G. Da San Martino, Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 9th International Conference of the CLEF Association (CLEF 2018), Springer, 2018, pp. 372–387. find
  8. A. Barrón-Cedeño, T. Elsayed, P. Nakov, G. Da San Martino, M. Hasanain, R. Suwaileh, F. Haouari, N. Babulkov, B. Hamdan, A. Nikolov, et al., Overview of CheckThat! 2020: Automatic identification and verification of claims in social media, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), Springer, 2020, pp. 215–236. find
  9. A. Barrón-Cedeño, F. Alam, J. M. Struß, P. Nakov, T. Chakraborty, T. Elsayed, P. Przybyła, T. Caselli, G. Da San Martino, F. Haouari, et al., Overview of the CLEF-2024 CheckThat! lab: Check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2024, pp. 28–52. find
  10. F. Alam, J. M. Struß, T. Chakraborty, S. Dietze, S. Hafid, K. Korre, A. Muti, P. Nakov, F. Ruggeri, S. Schellhammer, V. Setty, M. Sundriyal, K. Todorov, V. Venktesh, Overview of the CLEF-2025 CheckThat! lab: Subjectivity, fact-checking, claim normalization, and retrieval, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixteenth International Conference of the CLEF Association (CLEF 2025), 2025. find
  11. V. Venktesh, A. Anand, A. Anand, V. Setty, QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims, in: 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024), Association for Computing Machinery, 2024, pp. 650–660. find
  12. V. V, V. Setty, P. Chungkham, A. Anand, F. Alam, Overview of the CLEF-2026 CheckThat! lab task 2 on fact-checking numerical claims, 2026. find
  13. M. Liang, Y. Li, J. Xu, K. Asadi, X. Liu, S. Gu, K. Rangadurai, F. Shyu, S. Wang, S. Yang, Z. Li, J. Liu, M. Sun, F. Tian, X. Wei, C. Sun, J. Tao, S. Mei, W. Chen, S. Kolay, S. Pandey, H. Firooz, L. Simon, Generative reasoning re-ranker, arXiv preprint arXiv:2602.07774 (2026). ↗
  14. M. Wanner, S. Ebner, Z. Jiang, M. Dredze, B. Van Durme, A closer look at claim decomposition, in: Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 153–175. URL: https://aclanthology.org/2024.starsem-1.13/. doi: 10.18653/v1/2024.starsem-1.13 . ↗
  15. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, H. Hajishirzi, FActScore: Fine-grained atomic evaluation of factual precision in long form text generation, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 12076–12100. URL: https://aclanthology.org/2023.emnlp-main.741/. doi: 10.18653/v1/2023. emnlp-main.741 . ↗ our card
  16. R. Kamoi, T. Goyal, J. Diego Rodriguez, G. Durrett, WiCE: Real-world entailment for claims in Wikipedia, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 7561–7583. URL: https://aclanthology.org/2023.emnlp-main.470/. doi: 10.18653/v1/ 2023.emnlp-main.470 . ↗ our card
  17. K. Mitra, D. Zhang, S. Rahman, E. Hruschka, FactLens: Benchmarking fine-grained fact verification, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025, Association for Computational Linguistics, Vienna, Austria, 2025, pp. 18085–18096. URL: https://aclanthology.org/2025.findings-acl.929/. doi: 10.18653/v1/2025.findings-acl.929 . ↗
  18. W. Antoun, F. Baly, H. Hajj, Arabert: Transformer-based model for arabic language understanding, arXiv preprint arXiv:2003.00104 (2020). URL: https://arxiv.org/abs/2003.00104. doi: 10.48550/ arXiv.2003.00104 . ↗ our card
  19. A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, Z. Zhang, Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, arXiv preprint arXiv:2409.12122 (2024). ↗ our card
  20. PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www.pace.gatech.edu. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sinha-2026-ds-gt-arc-checkthat-2026/