Elamur Research⌕
☰
home / papers / 2018 / ClaimRank: Detecting Check-Worthy Claims in Arabic and English

ClaimRank: Detecting Check-Worthy Claims in Arabic and English

Paper · 2018 · Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeño et al.

Summary

ClaimRank addresses the first stage of fact-checking: deciding which sentences in a text deserve verification. Manual checking is slow, so the system ranks sentences by how check-worthy they appear and lets journalists start with the most promising ones. Its training labels reflect which statements from US political debates nine verification outlets actually picked to examine: The Guardian, ABC, PolitiFact, CNN, Washington Post, FactCheck, NYT, NPR and Chicago Tribune. Users can therefore emulate one outlet's selection habits or the combined pool. Each sentence gets a rich feature set (lexical, syntactic, sentiment, lexicon, structural, topic and embedding features), and a small feed-forward network scores it. The system was extended from English to Arabic by translating two debates and using cross-language word embeddings. In the reported experiments, Arabic ranking quality comes close to English, and every configuration clearly beats a random baseline. The deployed tool is a web demo that handles arbitrary text such as news or interviews, not only debates. The authors describe it as the only check-worthiness system available for Arabic.

Key points

  • Data: seven English political debates plus Arabic translations of two of them. For evaluation, training used five English debates and testing used the remaining two, in English or in Arabic translation.
  • Model: two ReLU hidden layers (200 and 50 units) and a sigmoid output, trained with SGD for 100 iterations. Sentences are ranked by the probability of the positive class.
  • Features were reused from ClaimBuster and Gencheva et al. (2017): TF.IDF words, POS tags, named entities, sentiment, length, bias/assertiveness/subjectivity lexicons, position, LDA topics and embeddings. Speaker metadata and discourse features were dropped so the system works on any text and on Arabic.
  • English MAP: word2vec 0.323, VecMap 0.298, MUSE 0.319, Attract-Repel 0.342, random 0.161. Arabic MAP: VecMap 0.291, MUSE 0.302, Attract-Repel 0.263. MUSE was chosen for the final system because it was strongest overall, including on P@20 and P@50.
  • Arabic pipeline: custom regex sentence splitting, Farasa for segmentation and POS tagging, and POS tags of both languages mapped to the Universal tagset.

Interesting ideas

  • I1. Instead of collapsing the nine organizations into one gold standard, the system keeps each source's labels separate, so it can reproduce any single annotator's policy or their union; this makes disagreement usable instead of noise.
  • I2. Evaluation uses precision at cutoffs matching a realistic human budget (P@20, P@50), since most outlets checked at most 50 claims per debate; the metric reflects how the output is actually consumed.
  • I3. Tagsets from different taggers (Penn Treebank, Farasa) were mapped into the Universal tagset so that one feature space serves both languages. A cheap normalization layer like this lets downstream components stay unchanged across sources.
  • I4. Attract-Repel embeddings did poorly on Arabic, likely because they were trained on unsegmented text where attached clitics inflate the vocabulary. This shows that segmentation choices upstream decide the quality of lexical resources downstream.
  • I5. Features tied to the original genre (speaker identity, system messages) were deliberately removed, trading some accuracy for the ability to process free text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver data could keep per-witness labels, so we can train or inspect models that imitate each parser separately as well as their agreement, and see where the witnesses really diverge.
  • I2 (answer): when Mova proposes candidate answers or inferences for human review, scoring precision within the top few items would show whether a reviewer's limited attention is spent well.
  • I3 (tree): converting every treebank's tags and UD dialect into one canonical inventory before parsing or feature extraction lets the perceptron parser and gparse share training data without per-source special cases.
  • I4 (tokens): splitting contractions and clitic-like pieces consistently in the en crate before building lemma or embedding tables should reduce sparsity and keep lexicon lookups aligned with the UD tokens.
  • I5 (frames): frame and speech-act rules in prag should avoid relying on genre-specific cues (dialogue markup, speaker tags) unless they are present, so the same rules carry over to fables, news and school texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🏢AmazonSeattle · United States112 works · 40,565 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria21 works · 2,595 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 works

Cite

Cite as Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeño, Lluís Màrquez and Preslav Nakov. 2018. ClaimRank: Detecting Check-Worthy Claims in Arabic and English. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-5006

@inproceedings{jaradat2018claimrank,
  title = {{ClaimRank: Detecting Check-Worthy Claims in Arabic and English}},
  author = {Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeño, Lluís Màrquez and Preslav Nakov},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations},
  year = {2018},
  doi = {10.18653/v1/n18-5006},
  url = {https://doi.org/10.18653/v1/n18-5006}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. ↗
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT ’16, pages 11–16, San Diego, CA, USA.
  3. Bing Liu, Minqing Hu, and Junsheng Cheng. 2005. Opinion observer: Analyzing and comparing opinions on the web. In Proceedings of the 14th International Conference on World Wide Web, WWW ’05, pages 342–351, New York, NY, USA. find
  4. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 451–462, Vancouver, Canada.
  5. Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. In Proceedings of the Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistic, ETMTNLP ’02, pages 63–70, Philadelphia, PA, USA. ↗
  6. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python, 1st edition. O’Reilly Media, Inc. find
  7. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330. ↗
  8. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993–1022. ↗
  9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, NIPS ’13, pages 3111–3119, Lake Tahoe, CA, USA. ↗
  10. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16th Conference of the European Association for Machine Translation, EAMT ’12, pages 261–268, Trento, Italy. find
  11. Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309–324. ↗
  12. Pepa Gencheva, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP ’17, pages 267–276, Varna, Bulgaria. find
  13. Ayush Patwari, Dan Goldwasser, and Saurabh Bagchi. 2017. TATHYA: A multi-classifier system for detecting check-worthy statements in political debates. In Proceedings of the ACM on Conference on Information and Knowledge Management, CIKM ’17, pages 2259–2262, Singapore. find
  14. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR ’15, pages 315–323, Fort Lauderdale, FL, USA. find
  15. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eight International Conference on Language Resources and Evaluation, LREC ’12, pages 2089– 2096, Istanbul, Turkey.
  16. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM ’15, pages 1835– 1838, Melbourne, Australia. find
  17. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL ’13, pages 1650–1659, Sofia, Bulgaria.
  18. Joan B. Hooper. 1974. On Assertive Predicates. Indiana University Linguistics Club. Indiana University Linguistics Club. ↗
  19. Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Comput. Linguist., 41(3):385–435. ↗
  20. Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, and Anna Korhonen. 2017. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 56–68, Vancouver, Canada. find
  21. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jaradat-2018-claimrank-detecting-check-worthy-claims/