Elamur Research⌕
☰
home / papers / 2018 / Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness

Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness

Paper · 2018 · Pepa Atanasova, Alberto Barron-Cedeno, Tamer Elsayed et al.

Summary

The paper reports on Task 1 of the CLEF-2018 CheckThat! Lab. Fact-checkers can only review a small share of what politicians say, so the task asks systems to rank the sentences of a debate or speech by how much they deserve verification. The organizers built the CT-CWC-18 corpus from 2016 US Presidential campaign debates and later speeches by Donald Trump. A sentence was labelled positive when FactCheck.org had questioned the claim it contains. The labels were then corrected by hand: boundaries were adjusted and repeated mentions of the same claim were added. Part of the material was translated into Arabic. The task was framed as ranking and scored with MAP. Thirty teams registered and seven submitted runs. The strongest systems combined recurrent or multi-layer neural networks with distributional embeddings, lexicon matching and syntactic dependency features. They reached a MAP of 0.18 for English and 0.15 for Arabic, which leaves a lot of room for improvement. The organizers released all data and scoring scripts to support further work on check-worthiness.

Key points

  • CT-CWC-18 (English) contains five debates and five speeches: 8,946 sentences, of which only 282 are check-worthy. The Arabic part has 7,254 sentences and 205 check-worthy ones.
  • The first three debates were used for training. Everything else, including all speeches, was held out for testing.
  • MAP, macro-averaged over test documents, was the official measure. MRR, mean R-Precision and mean P@k were also reported.
  • External fact-checking annotations were not allowed. A URL-checking script was provided to keep material from fact-checking sites out of systems.
  • Systems scored higher on speeches than on debates. A min-max-normalized score-sum ensemble of all teams gave a small MAP gain over the best individual system. Leaving out the team "blue" made the ensemble even better.

Interesting ideas

  • I1. Labels taken from external commentary were re-aligned by hand: claims were moved to the right neighboring sentence and every repeat of a claim was marked, not just the one the experts commented on.
  • I2. The ensemble added up min-max-normalized system scores, then dropped each member in turn; this exposed a component that was hurting the result, while summing reciprocal ranks performed much worse.
  • I3. Organizers supplied a leakage checker so that no answers could be pulled from fact-checking websites.
  • I4. One team translated English lexicon entries into Arabic one by one and did worse than a team that translated full sentences; the authors attribute this to lost context and to translation direction.
  • I5. Prior work cited in the paper found that the context of a sentence, such as its position within a long turn and who is speaking, helps predict whether it is worth checking.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when silver labels are derived from outside sources, a propagation step could copy each label to all coreferent restatements, so near-identical sentences do not end up as false negatives.
  • I2 (learn): witness-parser agreement could be scored on normalized confidences, with routine leave-one-witness-out runs to find witnesses that degrade the silver data.
  • I3 (learn): evaluations on fables or school tests could include a similar filter that rejects knowledge sources containing the test answers, serving as a built-in negative control.
  • I4 (back): translation through the meta-world should work on whole frames with their context rather than isolated lexical items, which supports translating at graph level.
  • I5 (frames): speech-act tagging could combine speaker and discourse-position features to separate verifiable factual assertions from opinions, giving an explainable "claim" act.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria21 works · 2,595 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Pepa Atanasova, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness. arXiv:1808.05542 · CLEF-2018.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.05542 · arXiv 1808.05542

@misc{atanasova2018overview,
  title = {{Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness}},
  author = {Pepa Atanasova, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni {Da San Martino} and Preslav Nakov},
  year = {2018},
  doi = {10.48550/arxiv.1808.05542},
  eprint = {1808.05542},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.05542}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agez, R., Bosc, C., Lespagnol, C., Mothe, J., Petitcol, N.: IRIT at CheckThat! 2018. In: Cappellato et al. [3] find
  2. Barrón-Cedeño, A., Elsayed, T., Suwaileh, R., Màrquez, L., Atanasova, P., Zaghouani, W., Kyuchukov, S., Da San Martino, G., Nakov, P.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 2: Factuality. In: Cappellato et al. [3] find
  3. Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): Working Notes of CLEF 2018–Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France (2018) find
  4. Castillo, C., Mendoza, M., Poblete, B.: Information credibility on Twitter. In: Proceedings of the 20th International Conference on World Wide Web. pp. 675– 684. WWW ’11, Hyderabad, India (2011) find
  5. Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Wong Sak Hoi, G., Zubiaga, A.: SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In: Proceedings of the 11th International Workshop on Semantic Evaluation. pp. 60–67. SemEval ’17, Vancouver, Canada (2017)
  6. Friedman, J.H.: Greedy function approximation: A gradient boosting machine. The Annals of Statistics 29(5), 1189–1232 (2001) find
  7. Fuhr, N., Nejdl, W., Peters, I., Stein, B., Giachanou, A., Grefenstette, G., Gurevych, I., Hanselowski, A., Jarvelin, K., Jones, R., Liu, Y., Mothe, J.: An Information Nutritional Label for Online Documents. ACM SIGIR Forum 51, 46–66 (2018) find
  8. Gencheva, P., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: A context-aware approach for detecting worth-checking claims in political debates. In: Proceedings of the International Conference Recent Advances in Natural Language Processing. pp. 267–276. RANLP ’17, Varna, Bulgaria (2017) find
  9. Granados, A., Cebrian, M., Camacho, D., de Borja Rodriguez, F.: Reducing the loss of information through annealing text distortion. IEEE Transactions on Knowledge and Data Engineering 23(7), 1090–1102 (2011) find
  10. Hansen, C., Hansen, C., Simonsen, J., Lioma, C.: The Copenhagen Team Participation in the Check-Worthiness Task of the Competition of Automatic Identification and Verification of Claims in Political Debates of the CLEF-2018 Fact Checking Lab. In: Cappellato et al. [3] find
  11. Hassan, N., Li, C., Tremayne, M.: Detecting check-worthy factual claims in presidential debates. In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management. pp. 1835–1838. CIKM ’15, Melbourne, Australia (2015) Overview of the CLEF-2018 CheckThat! Lab. Task 1: Check-Worthiness find
  12. Hassan, N., Tremayne, M., Arslan, F., Li, C.: Comparing automated factual claim detection against judgments of journalism organizations. In: Computation + Journalism Symposium. Stanford, CA, USA (2016) find
  13. Jaradat, I., Gencheva, P., Barrón-Cedeño, A., Màrquez, L., Nakov, P.: ClaimRank: Detecting check-worthy claims in Arabic and English. In: Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics. NAACL-HLT ’18, New Orleans, LA, USA (2018)
  14. Karadzhov, G., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: Fully automated fact checking using external sources. In: Proceedings of the Conference on Recent Advances in Natural Language Processing. pp. 344–353. RANLP ’17, Varna, Bulgaria (2017) find
  15. Mikolov, T., Yih, W.t., Zweig, G.: Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 746–751. NAACL-HLT ’13, Atlanta, GA, USA (2013)
  16. Nakov, P., Barrón-Cedeño, A., Elsayed, T., Suwaileh, R., Màrquez, L., Zaghouani, W., Atanasova, P., Kyuchukov, S., Da San Martino, G.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. In: Bellot, P., Trabelsi, C., Mothe, J., Murtagh, F., Nie, J., Soulier, L., Sanjuan, E., Cappellato, L., Ferro, N. (eds.) Proceedings of the Ninth International Conference of the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Lecture Notes in Computer Science, Springer, Avignon, France (2018) find
  17. Papadopoulos, S., Bontcheva, K., Jaho, E., Lupu, M., Castillo, C.: Overview of the special issue on trust and veracity of information in social media. ACM Trans. Inf. Syst. 34(3), 14:1–14:5 (Apr 2016) find
  18. Patwari, A., Goldwasser, D., Bagchi, S.: TATHYA: a multi-classifier system for detecting check-worthy statements in political debates. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. pp. 2259–2262. CIKM ’17, Singapore (2017) find
  19. Popat, K., Mukherjee, S., Strötgen, J., Weikum, G.: Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In: Proceedings of the 26th International Conference on World Wide Web Companion. pp. 1003– 1012. WWW ’17, Perth, Australia (2017) find
  20. Rashkin, H., Choi, E., Jang, J.Y., Volkova, S., Choi, Y.: Truth of varying shades: Analyzing language in fake news and political fact-checking. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 2931–2937. EMNLP ’17 (2017)
  21. Shiralkar, P., Flammini, A., Menczer, F., Ciampaglia, G.L.: Finding streams in knowledge graphs to support fact checking. In: Proceedings of the IEEE International Conference on Data Mining. ICDM ’17, New Orleans, LA, USA (2017) find
  22. Vlachos, A., Riedel, S.: Fact checking: Task definition and dataset construction. In: Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. pp. 18–22. Baltimore, MD, USA (2014)
  23. Yasser, K., Kutlu, M., , Elsayed, T.: bigIR at CLEF 2018: Detection and Verifi cation of Check-Worthy Political Claims. In: Cappellato et al. [3] find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/atanasova-2018-overview-clef-2018-checkthat-lab/