Elamur Research⌕
☰
home / papers / 2024 / IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection

IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection

Summary

The paper describes the University of Stavanger IAI group's entry to the CLEF 2024 CheckThat! Lab Task 1. The task is a binary decision: does a sentence deserve fact-checking? The data cover English political debate transcripts and Dutch and Arabic tweets. The team compared two families of models. The first was fine-tuned encoder models (RoBERTa-Large, XLM-RoBERTa). The second was generative LLMs (GPT-3.5, GPT-4, Mistral 7b), used either through few-shot chain-of-thought prompting or through fine-tuning. They also tested several data tricks:

  • cross-lingual training with machine-translated data;
  • tweet normalization;
  • LLM-based style transfer from debate style to tweet style.

Their final scores were:

  • Arabic: first place, by translating the Arabic test set into English and labelling it with an English-fine-tuned GPT-3.5.
  • Dutch: third place, with GPT-4 prompting.
  • English: ninth place, with a plainly fine-tuned RoBERTa-Large.

Every system scored much lower on the hidden test set than on the development test split. The paper is useful mainly as an honest record of which augmentation choices helped, which did not, and how unreliable development estimates can be under domain shift.

Key points

  • Training set sizes were highly unequal and skewed toward the "No" class:
    • English: 22,501 sentences
    • Arabic: 7,333
    • Dutch: 995
  • Final F1 scores:
    • English: 0.753 (RoBERTa-Large on the untouched organizer data)
    • Dutch: 0.718 (GPT-4 with few-shot CoT)
    • Arabic: 0.569 (translate to English, then fine-tuned GPT-3.5)
  • A better Dutch run went unsubmitted. A GPT-3.5 fine-tuned on Spanish, Arabic and Dutch data reached 0.781 on the Dutch test set. That would have topped the leaderboard.
  • Translation tool: Google Translate, accessed through the deep-translator library, moved data between English, Dutch, Arabic and Spanish.
  • Two preprocessing steps did not help:
    • TweetNormalizer lowered results.
    • Style transfer with gpt-3.5-turbo-0125 (debate sentences rewritten as Arabic-style tweets) gave no gain.
  • A misleading self-check: the team hand-labelled 10% of the Arabic test data before gold labels were released. The chosen model scored 0.848 F1 on that sample but only 0.569 on the full set.

Interesting ideas

  • I1. Instead of building a strong classifier in a weaker language, translate the input into the language where the best model already exists and classify it there; this reuses one well-trained component across languages.
  • I2. The authors suspect TweetNormalizer hurt because it collapsed every distinct @username into a generic @USER token, deleting identifiers that carried signal; normalization can silently throw away useful information.
  • I3. Rewriting training data with an LLM so its genre matches the target domain (debate prose into tweets) is a cheap way to make in-domain training data, but here it gave no benefit, so such data must be tested before it is trusted.
  • I4. A small hand-labelled sample of the test set picked the "best" model, yet the full test set reversed the picture; small in-house checks under distribution shift can mislead badly.
  • I5. The paper recommends choosing checkpoints by macro-averaged F1 rather than accuracy, so the rare positive class counts as much as the dominant one in imbalanced data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Non-English input could be routed through translation into English, so the existing English reader handles meaning, with the translation step logged to keep the chain explainable.
  • I2 (tokens): The tokenizer should keep handles, names and similar identifiers as distinct, typed tokens rather than flattening them, so later stages can still link them to entities.
  • I3 (tree): Restyled treebank sentences could serve as candidate silver data for gparse on new genres, but they should be admitted only after an A/B comparison against the plain data.
  • I4 (learn): Evaluations should report how stable rankings are across several held-out slices and include negative controls, rather than trusting a single small sample.
  • I5 (learn): When training detectors for rare phenomena (for example irony or particular speech acts), select models by macro-F1 so frequent labels do not mask failures on rare ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Semitic languages

 Institutions

Also on the byline: University of Stavanger.

Cite

Cite as Peter R\oysland Aarnes, Vinay Setty and Petra Galuščakova. 2024. IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection. arXiv:2408.01118.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.01118 · arXiv 2408.01118

@misc{aarnes2024iai,
  title = {{IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection}},
  author = {Peter Roysland Aarnes, Vinay Setty and Petra Galuvsvcakova},
  year = {2024},
  doi = {10.48550/arxiv.2408.01118},
  eprint = {2408.01118},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.01118}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Konstantinovskiy, O. Price, M. Babakar, A. Zubiaga, Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection, Digital Threats 2 (2021) 1–16. find
  2. N. Hassan, F. Arslan, C. Li, M. Tremayne, Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017, pp. 1803–1812. find
  3. F. Alam, S. Shaar, F. Dalvi, H. Sajjad, A. Nikolov, H. Mubarak, G. Da San Martino, A. Abdelali, N. Durrani, K. Darwish, A. Al-Homaid, W. Zaghouani, T. Caselli, G. Danoe, F. Stolk, B. Bruntink, P. Nakov, Fighting the covid-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society, in: Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 611–649. find
  4. A. Barrón-Cedeño, F. Alam, T. Chakraborty, T. Elsayed, P. Nakov, P. Przybyła, J. M. Struß, F. Haouari, M. Hasanain, F. Ruggeri, X. Song, R. Suwaileh, The CLEF-2024 CheckThat! Lab: Check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness, in: Advances in Information Retrieval, 2024, pp. 449–458. find
  5. I. Siles, P. J. Boczkowski, Making sense of the newspaper crisis: A critical assessment of existing research and an agenda for future work, New Media & Society 14 (2012) 1375–1394. find
  6. S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, Science 359 (2018) 1146–1151. find
  7. H. Allcott, M. Gentzkow, Social media and fake news in the 2016 election, Journal of Economic Perspectives 31 (2017) 211–236. find
  8. A. Bovet, H. A. Makse, Influence of fake news in twitter during the 2016 us presidential election, Nature Communications 10 (2019) 7. find
  9. T. Pavlov, G. Mirceva, Covid-19 fake news detection by using bert and roberta models, in: 2022 45th jubilee international convention on information, communication and electronic technology (MIPRO), 2022, pp. 312–316. find
  10. Z. Guo, M. Schlichtkrull, A. Vlachos, A survey on automated fact-checking, Transactions of the Association for Computational Linguistics 10 (2022) 178–206. our card
  11. F. Arslan, N. Hassan, C. Li, M. Tremayne, A benchmark dataset of check-worthy factual claims, 2020. find
  12. T. Elsayed, P. Nakov, A. Barrón-Cedeño, M. Hasanain, R. Suwaileh, G. D. S. Martino, P. Atanasova, Overview of the clef-2019 checkthat!: Automatic identification and verification of claims, 2019. our card
  13. A. Gupta, V. Srikumar, X-fact: A new benchmark dataset for multilingual fact checking, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2021, pp. 675–682. find
  14. X. Zeng, A. S. Abumansour, A. Zubiaga, Automated fact-checking: A survey, 2021. find
  15. P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, H. Mubarak, A. Nikolov, Y. S. Kartal, J. Beltrán, Overview of the CLEF-2022 CheckThat! lab task 1 on identifying relevant claims in tweets, in: Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, 2022. find
  16. F. Alam, A. Barrón-Cedeño, G. S. Cheema, G. K. Shahi, S. Hakimov, M. Hasanain, C. Li, R. Míguez, H. Mubarak, W. Zaghouani, P. Nakov, Overview of the clef-2023 checkthat! lab task 1 on check-worthiness of multimodal and multigenre content (2023). find
  17. S. K. N. Mingzhe Du, Sujatha Das Gollapalli, NUS-IDS at CheckThat! 2022: identifying check-worthiness of tweets using CheckthaT5, in: N. Faggioli, Guglielmo andd Ferro, A. Hanbury, M. Potthast (Eds.), Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF 2022, 2022. find
  18. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Raffel, mt5: A massively multilingual pre-trained text-to-text transformer, 2021. our card
  19. M. Sawinski, K. Węcel, E. Księżniak, M. Stróżyna, W. Lewoniewski, P. Stolarski, W. Abramowicz, Openfact at checkthat! 2023: Head-to-head gpt vs. bert - a comparative study of transformers language models for the detection of check-worthy claims, CLEF 2023, 2023. find
  20. M. Abdul-Mageed, A. Elmadany, E. M. B. Nagoudi, Arbert & marbert: Deep bidirectional transformers for arabic, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 7088–7105. our card
  21. H. T. Sadouk, F. Sebbak, H. E. Zekiri, Es-vrai at checkthat! 2023: Analyzing checkworthiness in multimodal and multigenre contents through fusion and sampling approaches (2023). find
  22. S. Henning, W. Beluch, A. Fraser, A. Friedrich, A survey of methods for addressing class imbalance in deep-learning based natural language processing, in: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 523–540. find
  23. L. Biewald, Experiment tracking with weights and biases}, 2020. find
  24. D. Q. Nguyen, T. Vu, A. Tuan Nguyen, Bertweet: A pre-trained language model for english tweets, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 9–14. our card
  25. T. Kocmi, E. Avramidis, R. Bawden, O. Bojar, A. Dvorkovich, C. Federmann, M. Fishel, M. Freitag, T. Gowda, R. Grundkiewicz, B. Haddow, P. Koehn, B. Marie, C. Monz, M. Morishita, K. Murray, M. Nagata, T. Nakazawa, M. Popel, M. Popović, M. Shmatova, Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet, in: Proceedings of the Eighth Conference on Machine Translation, WMT’23, 2023, pp. 1–42. find
  26. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, . Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, . Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. d. A. B. Peres, M. Petrov, H. P. d. O. Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. J. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, B. Zoph, Gpt-4 technical report, 2024. our card
  27. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. find
  28. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, R. Lowe, Training language models to follow instructions with human feedback, 2022. our card
  29. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, 2020. find
  30. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, 2019. find
  31. A. J. Becker, Automated fact-checking of podcasts, 2023. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aarnes-2024-iai-group-checkthat-2024-transformer/