Elamur Research⌕
☰
home / papers / 2022 / ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection

ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection

Summary

The paper addresses three-way natural language inference for Arabic: deciding whether one sentence entails, contradicts, or is neutral toward another. The authors focus on contradiction. They argue that Arabic is low-resource for this task, since the only earlier Arabic RTE set had about 600 pairs and binary labels. They therefore built ArNLI semi-automatically. It contains more than 12k sentences (over 6000 pairs), labelled entailment, contradiction or neutral, and its contradictions cover the de Marneffe et al. typology. Their method follows Stanford's English contradiction work. A hand-built contradiction feature vector is joined to a text representation, such as word or character n-grams, bag-of-words, TF-IDF or word2vec. The combined vector is fed to classical classifiers. For comparison, the English PHEME and SICK sets were machine-translated into Arabic. Random Forest performed best, reaching 99% on translated PHEME, 60% on translated SICK and 75% on ArNLI. The released dataset gives Arabic NLI a public three-way benchmark and a baseline to beat.

Key points

  • ArNLI has more than 12k sentences, over 6000 pairs, and three classes. It is public at github.com/Khloud-AL/ArNLI.
  • Contradiction types follow the de Marneffe et al. typology: Antonym, Negation, Numeric, Factive, Structure, Lexical, WK.
  • Six classifiers were compared: SVM, SGD, Decision Tree, AdaBoost, KNN and Random Forest. Each was paired with several representations: BoW, word/char uni/bi/trigrams, TF-IDF (word, char, union), word2vec and word2vec-TF-IDF.
  • Random Forest accuracy was 0.99 on translated PHEME, 0.60 on translated SICK (SemEval2014 Task1) and 0.75 on ArNLI.
  • The best ArNLI setting joined the contradiction vector with either word trigrams or character bag-of-words.

Interesting ideas

  • I1. Tag each contradiction pair with its type from a fixed typology, such as negation, antonym, numeric mismatch, factive, structural, lexical or world knowledge. This lets errors be broken down by the kind of reasoning involved rather than reported as one score. — For Mova (answer): Contradiction and consistency probes could be labelled with the same typology, so failures can be traced to the responsible stage, e.g. numeric mismatches to measures and antonyms to WordNet knowledge.
  • I2. Keep a small, explicit contradiction feature block (negation, antonym and number cues) separate from the general text representation. This keeps the decisive signals inspectable. — For Mova (frames): Polarity, antonymy and quantity conflicts could be checked as explicit comparisons between aligned frames of two sentences, each with a recorded reason.
  • I3. The authors' table of translation failures shows machine translation can flip or blur meaning, for example changing negation or argument structure. Such failures can silently corrupt a translated benchmark's labels. — For Mova (back): Translation through the meta-world could be tested on NLI pairs, checking that the entailment or contradiction label still holds after the text → graph → text round trip.
  • I4. Related work (Harabagiu et al.) built contradiction data by negating entailment hypotheses. It also added negation markers to non-entailments, so that a negation cue alone would not predict the label. — For Mova (learn): Synthetic contradiction pairs could come with such negative controls, checking that a learner reasons about meaning rather than a surface negation word.
  • I5. Near-perfect scores on translated PHEME next to 0.60 on SICK suggest some benchmarks are much easier than others. Results should be compared across several sets before trusting them. — For Mova (learn): Evaluation could always include a harder, typologically mixed set beside easy ones, to expose shortcut gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

4 papers of our library cite it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.7494/csci.2023.24.2.4378 · arXiv 2209.13953

@misc{jallad2022arnli,
  title = {{ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection}},
  author = {Jallad, Khloud Al and Ghneim, Nada},
  year = {2022},
  doi = {10.7494/csci.2023.24.2.4378},
  eprint = {2209.13953},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.7494/csci.2023.24.2.4378}
}

References

52 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Romanov and C. Shivade, "Lessons from Natural Language Inference in the Clinical Domain," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels,Belgium, 2018. find our card
  2. M. Sammons, V. Vydiswaran and D. Roth, "Recognizing Textual Entailment," in Multilingual Natural Language Applications: From Theory to Practice, University of Illinois, 2012, pp. 209-258. find
  3. D. Roth, M. Sammons and V. G. Vydiswaran, "A framework for Entailed Relation Recognition," in ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics , Suntec, 2009. find
  4. A. Mishra, D. Patel, A. Vijayakumar, X. L. Li, P. Kapanipathi and K. Talamadupula, "Reading Comprehension as Natural Language Inference:A Semantic Analysis," in Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), Barcelona, Spain , December,2020. find
  5. P. Clark, "Recognizing Textual Entailment, QA4MRE, and Machine Reading," in {CLEF} 2012 Evaluation Labs and Workshop, Online Working Notes, Rome,Italy, September 17-20, 2012, vol. 1178, P. Forner, J. Karlgren and C. Womser{-}Hacker, Eds., Rome, Italy, CEUR-WS.org, 2012. find
  6. S. Padó, M. Galley, D. Jurafsky and C. D. Manning, "Robust Machine Translation Evaluation with Entailment Features," in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Suntec, Singapore, 2009. find
  7. A. Williams, N. Nangia and S. Bowman, "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference," in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, New Orleans, Louisiana, 2018. find
  8. R. Delmonte, A. Bristot, M. A. P. Boniforti and S. Tonelli, "Entailment and Anaphora Resolution in RTE3," Italy, 2007. find
  9. M.-C. d. Marneffe, A. N. Rafferty and C. D. Manning, "Finding Contradictions in Text," in Proceedings of ACL-08: HLT,, Columbus, Ohio, USA, 2008. find our card
  10. "Japanese Realistic Textual Entailment Corpus," in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), Marseille, 2020. find
  11. H. Hu, K. Richardson, L. Xu, L. Li, S. Kubler and L. S. Moss, "OCNLI: Original Chinese Natural Language Inference," arxiv, 2020. find
  12. G. Rocha and H. L. Cardoso, "Recognizing Textual Entailment: Challenges in the Portuguese Language," in Text Mining and Applications (TeMA) track of the 18th EPIA Conference on Artificial Intelligence (EPIA 2017) , Portugal, 2017. find
  13. K. Eichler, A. Gabryszak and G. Neumann, "An analysis of textual inference in German customer emails," in Proceedings of the Third Joint Conference on Lexical and Computational Semantics, 2014. find
  14. J. Bos, F. M. Zanzotto and M. Pennacchiotti, "Textual Entailment at EVALITA 2009," 2009. find
  15. E. R. Fonseca, L. B. d. Santos, M. Criscuolo and S. M. Aluisio, "Overview of the evaluation of semantic similarity and textual inference," in Proposta recebida em Setembro 2016 e aceite para publica c ̧ a ̃ o em Novembro 2016., 2016. find
  16. H. Amirkhani, M. A. Jafari, A. Amirak, Z. Pourjafari, S. F. Jahromi and Z. Kouhkan, "FarsTail: A Persian Natural Language Inference Dataset," arxiv, 2020. find
  17. E. Budur, R. . ̈ elik, T. G. o ̈ r and C. Potts, "Data and Representation for Turkish Natural Language Inference," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020. find
  18. S. Harabagiu, A. Hickl and F. Lacatusu, "Negation, Contrast and Contradiction in Text Processing," American Association for Artificial Intelligence , 2006. find
  19. B. MacCartney, T. Grenager, M.-C. deMarneffe, D. Cer and C. D.Manning, "Learning to recognize features of valid textual entailments," in In Proceedings of the North AmericanAssociation of Computational Linguistics (NAACL-06)., 2006. find
  20. A. Ritter, D. Downey, S. Soderland and O. Etzioni, "It’s a Contradiction—No, it’s Not: A Case Study using Functional Relations," in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing,, Honolulu, 2008. find
  21. A. Yates, M. Cafarella, M. Banko, O. Etzioni, M. Broadhead and S. Soderland, "TextRunner: Open Information Extraction on the Web," NAACL HLT Demonstration Program, pp. 25-26, 2007. find
  22. L. Li, B. Qin and T. Liu, "Contradiction Detection with Contradiction-Specific Word Embedding," algorithms, vol. 10, no. 59, 24 May 2017. find
  23. O.-M. Sulea, "Recognizing Textual Entailment in Twitter Using Word Embeddings," in Proceedings of the 2nd Workshop on Evaluating Vector-Space Representations for NLP, Copenhagen, Denmark, 2017. find
  24. V. Lingam, S. Bhuria, M. Nair, D. Gurpreetsingh, A. Goyal and A. Sureka, "Deep learning for conflicting statements detection in text," PeerJ Preprints, Mar 2018. find
  25. P. Lendvai, I. Augenstein, K. Bontcheva and T. Declerck, "Monolingual Social Media Datasets for Detecting Contradiction and Entailment," in In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), Portorož, Slovenia, 2016. find
  26. "Microsoft Research Paraphrase Corpus," Microsoft, 2005. [Online]. Available: https://www.microsoft.com/en-us/download/details.aspx?id=52398. [Accessed 2021]. ↗
  27. W. B. Dolan and C. Brockett, "Automatically Constructing a Corpus of Sentential Paraphrases," in Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. find our card
  28. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy and S. R. Bowman, "GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM FOR NATURAL LANGUAGE UNDERSTANDING," in ICLR 2019, 2019. find our card
  29. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy and S. R. Bowman, "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems," in NeurIPS 2019, 2019. find our card
  30. "superGlue," NYU; FaceBook; DeepMind;UWNLP, 2019. [Online]. Available: https://super.gluebenchmark.com/. [Accessed 2022]. ↗
  31. "ANli," allenai, 2019. [Online]. Available: https://leaderboard.allenai.org/anli/submissions/get-started. [Accessed 2022]. ↗
  32. K. Yuta and M. Christopher, "{C}ontract{NLI}: A Dataset for Document-level Natural Language Inference for Contracts," in Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, Association for Computational Linguistics, 2021, pp. 1907-1919. find
  33. S. Wang, H. Fang, M. Khabsa, H. Mao and H. Ma, "Entailment as Few-Shot Learner," arxiv, April 2021. find
  34. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and V. Stoyanov, "RoBERTa: A Robustly Optimized BERT Pretraining Approach," ArXiv, vol. abs/1907.11692, 2019. ↗
  35. P. He, X. Liu, J. Gao and W. Chen, "DeBERTa: Decoding-enhanced BERT with Disentangled Attention," in ICLR 2021 , 2021. find our card
  36. N. Almarwani and M. Diab, "Arabic Textual Entailment with Word Embeddings," in Proceedings of The Third Arabic Natural Language Processing Workshop (WANLP), Spain, 2017. find
  37. B.-S. Mabrouka, W. Bakaria and M. Nejia, "Classification and Analysis of Arabic Natural Language Inference Systems," in 4th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, 2020. find
  38. M. Khader, A. Awajan and A. Alkouz, "Textual Entailment for Arabic Language based on Lexical and Semantic Matching," International Journal of Computing & Information Sciences, vol. 12, no. 1, pp. 67-74, September 2016. find
  39. M. Alabbas, "ArbTE: Arabic Textual Entailment," in Proceedings of the Student Research Workshop associated with RANLP 2011, Hissar, Bulgaria, 2011. find
  40. M. Alabbas and A. Ramsay, "Natural Language Inference for Arabic Using Extended Tree Edit Distance with Subtrees," Journal of Artificial Intelligence Research, 2014. find
  41. F. T. AL-Khawaldeh, "A Study of the Effect of Resolving Negation and Sentiment Analysis in Recognizing Text Entailment for Arabic," World of Computer Science and Information Technology Journal (WCSIT), vol. 5, no. 7, p. 5, 2015. find our card
  42. T. Boudaa, M. E. Marouani and N. Enneya, "Alignment Based Approach for Arabic Textual Entailment," in Second International Conference on Intelligent Computing in Data Sciences (ICDS 2018), 2018. find
  43. M. Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini and R. Zamparelli, "SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment," in Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland, 2014. find our card
  44. D. Jurafsky and J. H. Martin, Speech and Language Processing: An introduction to natural language processing,computational linguistics, and speech recognition, Stanford University, 2021. find
  45. C. Sammut and G. I. Webb, Eds., Encyclopedia of Machine Learning, Boston: Springer, 2011. find
  46. Z. Liu, Y. Lin and M. Sun, Representation Learning for Natural Language Processing, Springer, 2020. find
  47. "SVM Classifier," scikit-learn.org, [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html. [Accessed 11 11 2020]. ↗
  48. "SGD," scikit-learn.org, [Online]. Available: https://scikit-learn.org/stable/modules/sgd.html. [Accessed 11 11 2020]. ↗
  49. "DecisionTreeClassifier," scikit-learn.org, [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html. [Accessed 11 11 2020]. ↗
  50. "AdaBoostClassifier," scikit-learn.or, [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html. [Accessed 11 11 2020]. ↗
  51. "KNN," scikit-learn.org, [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html. [Accessed 11 11 2020]. ↗
  52. "RandomForestClassifier.," scikit-learn.org, [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html. [Accessed 11 11 2020]. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jallad-2022-arnli-arabic-natural-language-inference/