Elamur Research⌕
☰
home / papers / 2026 / E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

Summary

Arabic has few reliable benchmarks for textual entailment and natural language inference. Most existing sets come from a single construction method: machine translation, hand-crafting, or one domain. The authors build E-CONAN by combining Arabic sentence pairs from four kinds of sources:

  • automatically translated pairs
  • human-checked translations
  • pairs written by hand from textbooks for learners of Arabic as a foreign language
  • news headlines containing rumours

The resource has two variants: a binary RTE set (E-CONAN-2) and a three-class NLI set (E-CONAN-3). They test it with zero-shot classification using multilingual pretrained models, several LLMs and the Arabic-specific MARBERT, and compare against ArNLI and XNLI. E-CONAN turns out harder than XNLI and easier than ArNLI, which the authors argue makes it a good probe of generalization. mDeBERTa is the strongest encoder. LLM errors cluster between neutral and contradiction, so merging those two classes raises accuracy sharply. Error analysis shows encoders are fooled by shared vocabulary and LLMs by familiar topics. The data is released on Hugging Face and covers Modern Standard Arabic only.

Key points

  • Dataset sizes and sources. E-CONAN-2 has 24,875 pairs and E-CONAN-3 has 18,875 pairs. Sources are ArNLI, the Arabic sections of SNLI and XNLI, and AnsStance; ArEntail is used only in the 2-way set.
  • Encoder results. mDeBERTa, fine-tuned on four machine-translated NLI sets, scored 71% on E-CONAN-3 and 86% on XNLI, beating all other encoders. The paper counts these encoders as 9 in one place and 10 in another.
  • LLM results. Gemma was the best LLM on E-CONAN-3 at 68%. When neutral and contradiction are folded into one class, Gemma reaches 95% and Qwen 94%.
  • Domain mix. E-CONAN-3 is about 35% news, 30% everyday text, 20% social media and 15% multi-genre. It still inherits translation and crowdsourcing biases from its source sets.
  • Error patterns and coverage. Encoders over-rely on lexical overlap, for example missing negation when wording is similar, while LLMs over-rely on topical familiarity. Dialects are left for future work.

Interesting ideas

  • I1. Building one benchmark from several construction paradigms (translated, human-verified, hand-written, news) tests robustness across text styles better than any single-paradigm set; its difficulty lands between the extremes. — For Mova (learn): Assemble Mova's evaluation suites from mixed-origin sources and report scores per source, so a gain on one style of text is not mistaken for general progress.
  • I2. Re-scoring the 3-way task as 2-way exposes where errors really sit: models detect entailment well but confuse "unrelated" with "contradicts". — For Mova (answer): Score Mova's readers both with and without the neutral/contradiction split, so entailment failures and contradiction-detection failures are tracked separately.
  • I3. The error taxonomy contrasts encoders fooled by shared vocabulary with LLMs fooled by familiar topics; it suggests building probe pairs that pull these cues apart. — For Mova (learn): Add negative-control pairs with high word overlap but negation, and pairs on familiar topics with unrelated claims, to check that Mova's verdicts come from its meaning graph rather than surface cues.
  • I4. The paper defines entailment, contradiction and neutrality through set relations, such as a trip to Paris being contained in a trip to France. This gives each label an explicit, checkable criterion. — For Mova (world): Decide entailment in the world of the text by testing containment through the ontology (Paris part-of France), making every inference label explainable.
  • I5. Short, controlled pairs taken from textbooks for foreign learners of a language serve as a clean, hand-crafted layer next to noisy translated data. — For Mova (answer): Mine English-as-a-foreign-language textbooks for simple entailment and contradiction pairs as a low-noise test layer for Mova's school-course readers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/access.2026.3732060 · arXiv 2609.11334

@misc{jallad2026e,
  title = {{E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets}},
  author = {Jallad, Khloud AL and Ghneim, Nada and Rebdawi, Ghaida},
  year = {2026},
  doi = {10.1109/access.2026.3732060},
  eprint = {2609.11334},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/access.2026.3732060}
}

References

54 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Giampiccolo, H. T. Dang, B. Magnini, I. Dagan, E. Cabrio, and W. B. Dolan, “The Fourth PASCAL Recognizing Textual Entailment Challenge,” in Text Analysis Conference, 2008. [Online]. Available: https://api.semanticscholar.org/CorpusID:12381965 ↗
  2. O. and M. B. Dagan Ido and Glickman, “The PASCAL Recognising Textual Entailment Challenge,” in Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, I. and M. B. and d’Alché-B. F. Quiñonero-Candela Joaquin and Dagan, Ed., Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 177–190. find
  3. R. Bar-Haim et al., “The Second PASCAL Recognising Textual Entailment Challenge,” 2006. [Online]. Available: https://api.semanticscholar.org/CorpusID:13385138 ↗
  4. D. Giampiccolo, B. Magnini, I. Dagan, and B. Dolan, “The Third PASCAL Recognizing Textual Entailment Challenge,” in Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, S. Sekine, K. Inui, I. Dagan, B. Dolan, D. Giampiccolo, and B. Magnini, Eds., Prague: Association for Computational Linguistics, Jun. 2007, pp. 1–9. [Online]. Available: https://aclanthology.org/W07-1401 ↗
  5. D. Giampiccolo, H. T. Dang, B. Magnini, I. Dagan, E. Cabrio, and W. B. Dolan, “The Fourth PASCAL Recognizing Textual Entailment Challenge,” in Text Analysis Conference, 2008. [Online]. Available: https://api.semanticscholar.org/CorpusID:12381965 ↗
  6. L. Bentivogli, B. Magnini, I. Dagan, H. T. Dang, and D. Giampiccolo, “The Fifth PASCAL Recognizing Textual Entailment Challenge,” in https://huggingface.co/datasets/KhloudJ/E-CONAN Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009, NIST, 2009. [Online]. Available: https://tac.nist.gov/publications/2009/additional.papers/RTE5_overvi ew.proceedings.pdf ↗
  7. L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo, “The Sixth PASCAL Recognizing Textual Entailment Challenge,” in Text Analysis Conference, 2009. [Online]. Available: https://api.semanticscholar.org/CorpusID:858065 ↗
  8. L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo, “The Seventh PASCAL Recognizing Textual Entailment Challenge,” Theory and Applications of Categories, 2011, [Online]. Available: https://api.semanticscholar.org/CorpusID:5791809 ↗
  9. H. J. Levesque, E. Davis, and L. Morgenstern, “The Winograd Schema Challenge,” in AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011. [Online]. Available: https://api.semanticscholar.org/CorpusID:15710851 ↗
  10. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, L. Màrquez, C. Callison-Burch, and J. Su, Eds., Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 632–642. doi: 10.18653/v1/D15-1075. ↗
  11. N. and B. S. Williams Adina and Nangia, “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, 2018, pp. 1112–1122. [Online]. Available: http://aclweb.org/anthology/N18-1101 ↗
  12. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,” in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355. doi: 10.18653/v1/W18-5446. ↗ our card
  13. A. Conneau et al., “XNLI: Evaluating Cross-lingual Sentence Representations,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium: Association for Computational Linguistics, Oct. 2018, pp. 2475–2485. doi: 10.18653/v1/D18-1269. ↗
  14. Ž. Agić and N. Schluter, “Baselines and Test Data for Cross-Lingual Inference,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, and T. Tokunaga, Eds., Miyazaki, Japan: European Language Resources Association (ELRA), May 2018. [Online]. Available: https://aclanthology.org/L18-1614 ↗
  15. A. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” ArXiv, vol. abs/2204.02311, 2022, [Online]. Available: https://api.semanticscholar.org/CorpusID:247951931 ↗
  16. Q. Zhong et al., “Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE,” ArXiv, vol. abs/2212.01853, 2022, [Online]. Available: https://api.semanticscholar.org/CorpusID:254246784 ↗
  17. Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” 2019. find
  18. Z. Zhang et al., “Semantics-aware BERT for Language Understanding,” in AAAI Conference on Artificial Intelligence, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:202539891 ↗
  19. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V Le, “XLNet: Generalized Autoregressive Pretraining for Language Understanding,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, Eds., Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2019/file/dc6a7e655 d7e5840e66733e9ee67cc69-Paper.pdf ↗ our card
  20. S. Soltan et al., “AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model,” ArXiv, vol. abs/2208.01448, 2022, [Online]. Available: https://api.semanticscholar.org/CorpusID:251253416 ↗
  21. S. Wu et al., “BloombergGPT: A Large Language Model for Finance,” ArXiv, vol. abs/2303.17564, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:257833842 ↗ our card
  22. S. Banerjee, A. Mahajan, A. Agarwal, and E. Singh, “First Train to Generate, then Generate to Train: UnitedSynT5 for Few-Shot NLI,” CoRR, vol. abs/2412.09263, 2024, doi: 10.48550/ARXIV.2412.09263. ↗
  23. L. Xue et al., “ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models,” Trans Assoc Comput Linguist, vol. 10, pp. 291–306, 2022, doi: 10.1162/tacl_a_00461. ↗
  24. H. W. Chung, T. Févry, H. Tsai, M. Johnson, and S. Ruder, “Rethinking embedding coupling in pre-trained language models,” ArXiv, vol. abs/2010.12821, 2020, [Online]. Available: https://api.semanticscholar.org/CorpusID:225067567 ↗ our card
  25. O. Shliazhko, A. Fenogenova, M. Tikhonova, A. Kozlova, V. Mikhailov, and T. Shavrina, “mGPT: Few-Shot Learners Go Multilingual,” Trans Assoc Comput Linguist, vol. 12, pp. 58–79, 2024, doi: 10.1162/tacl_a_00633. ↗ our card
  26. P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding-enhanced BERT with Disentangled Attention,” 2021. find our card
  27. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “SpanBERT: Improving Pre-training by Representing and Predicting Spans,” Trans Assoc Comput Linguist, vol. 8, pp. 64–77, 2020, doi: 10.1162/tacl_a_00300. ↗ our card
  28. F. Iandola, A. Shaw, R. Krishna, and K. Keutzer, “SqueezeBERT: What can computer vision teach NLP about efficient neural networks?,” in Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, N. S. Moosavi, A. Fan, V. Shwartz, G. Glavaš, S. Joty, A. Wang, and T. Wolf, Eds., Online: Association for Computational Linguistics, Nov. 2020, pp. 124–135. doi: 10.18653/v1/2020.sustainlp-1.17. ↗
  29. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” ArXiv, vol. abs/1910.01108, 2019, [Online]. Available: https://api.semanticscholar.org/CorpusID:203626972 ↗
  30. M. Alabbas, “A Dataset for Arabic Textual Entailment,” in Proceedings of the Student Research Workshop associated with RANLP 2013, I. Temnikova, I. Nikolova, and N. Konstantinova, Eds., Hissar, Bulgaria: INCOMA Ltd. Shoumen, BULGARIA, Sep. 2013, pp. 7–13. [Online]. Available: https://aclanthology.org/R13-2002 ↗
  31. R. Obeidat, Y. Al-Harahsheh, M. Al-Ayyoub, and M. Gharaibeh, “ArEntail: manually-curated Arabic natural language inference dataset from news headlines,” Lang Resour Eval, 2024, doi: 10.1007/s10579-024-09731-1. ↗
  32. K. Al Jallad and N. Ghneim, “ARNLI: ARABIC NATURAL LANGUAGE INFERENCE ENTAILMENT AND CONTRADICTION DETECTION,” Computer Science, vol. 24, no. 2, Mar. 2023, doi: 10.7494/csci.2023.24.2.4378. ↗ our card
  33. W. Yin, J. Hay, and D. Roth, “Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3914–3923. doi: 10.18653/v1/D19-1404. ↗ our card
  34. M. Lewis et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds., Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880. doi: 10.18653/v1/2020.acl-main.703. ↗ our card
  35. Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” ArXiv, vol. abs/1907.11692, 2019, [Online]. Available: https://api.semanticscholar.org/CorpusID:198953378 ↗
  36. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers,” ArXiv, vol. abs/2002.10957, 2020, [Online]. Available: https://api.semanticscholar.org/CorpusID:211296536 ↗
  37. P. He, J. Gao, and W. Chen, “DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing,” ArXiv, vol. abs/2111.09543, 2021, [Online]. Available: https://api.semanticscholar.org/CorpusID:244346093 ↗ our card
  38. M. Laurer, W. van Atteveldt, A. Casas, and K. Welbers, “Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and BERT-NLI,” Political Analysis, vol. 32, no. 1, pp. 84–100, Jan. 2024, doi: 10.1017/pan.2023.20. ↗
  39. M. Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini, and R. Zamparelli, “SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment,” in Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), P. Nakov and T. Zesch, Eds., Dublin, Ireland: Association for Computational Linguistics, Aug. 2014, pp. 1–8. doi: 10.3115/v1/S14-2001. ↗ our card
  40. P. Lendvai, I. Augenstein, K. Bontcheva, and T. Declerck, “Monolingual Social Media Datasets for Detecting Contradiction and Entailment,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds., Portorož, Slovenia: European Language Resources Association (ELRA), May 2016, pp. 4602–4605. [Online]. Available: https://aclanthology.org/L16-1729 ↗
  41. M.-C. de Marneffe, A. N. Rafferty, and C. D. Manning, “Finding Contradictions in Text,” in Proceedings of ACL-08: HLT, J. D. Moore, S. Teufel, J. Allan, and S. Furui, Eds., Columbus, Ohio: Association for Computational Linguistics, Jun. 2008, pp. 1039–1047. [Online]. Available: https://aclanthology.org/P08-1118 ↗ our card
  42. A. Conneau et al., “XNLI: Evaluating Cross-lingual Sentence Representations,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds., Brussels, Belgium: Association for Computational Linguistics, Oct. 2018, pp. 2475–2485. doi: 10.18653/v1/D18-1269. ↗
  43. J. Khouja, “Stance Prediction and Claim Verification: An Arabic Perspective,” in Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), C. Christodoulopoulos, J. Thorne, A. Vlachos, O. Cocarascu, and A. Mittal, Eds., Online: Association for Computational Linguistics, Jul. 2020, pp. 8–17. doi: 10.18653/v1/2020.fever-1.2. ↗
  44. G. Wenzek et al., “CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data,” in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds., Marseille, France: European Language Resources Association, May 2020, pp. 4003–4012. [Online]. Available: https://aclanthology.org/2020.lrec-1.494 ↗ our card
  45. A. Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds., Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. doi: 10.18653/v1/2020.acl-main.747. ↗
  46. M. S. Bari et al., “ALLaM: Large Language Models for Arabic and English,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=MscdsFVZrN ↗ our card
  47. A. Yang et al., “Qwen2 Technical Report,” arXiv preprint arXiv:2407.10671, 2024. ↗
  48. A. Yang et al., “Qwen2.5 Technical Report,” arXiv preprint arXiv:2412.15115, 2024. ↗
  49. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual Instruction Tuning,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., Curran Associates, Inc., 2023, pp. 34892–34916. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/6dcf277ea 32ce3288914faf369fe6de0-Paper-Conference.pdf ↗ our card
  50. H. Liu et al., “LLaVA-NeXT: Improved reasoning, OCR, and world knowledge,” Jan. 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/ ↗
  51. H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” 2023, arXiv:2310.03744. ↗ our card
  52. G. Team, “Gemma 3,” 2025, [Online]. Available: https://goo.gle/Gemma3Report ↗
  53. Y. Alnumay et al., “Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM,” 2025. [Online]. Available: https://arxiv.org/abs/2503.14603 ↗
  54. DeepSeek-AI et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948 Khloud Al Jallad received the B.S. degree in informatics engineering from Arab International University, Syria, in 2014, and the M.D. degree in Big Data from Higher Institute for Applied Sciences and Technology (HIAST), Syria, in 2019. Currently a PhD candidate at HIAST, Syria. From 2017 till now, she has been working as a researcher and lecturer. Her research interests include NLP, DL, ML. Nada Ghneim received the Information Technology Engineering degree from Higher Institute for Applied Sciences and Technology (HIAST), Damascus, Syria, 1991, and the Postgraduate Degree (DEA) in Artificial Intelligence (Image, Robotics, Vision), from the National High School of Computer Science and Applied Mathematics in Grenoble (ENSIMAG), France, 1993, and a PhD degree in Language Sciences (Speech Communication) from “Institut de la Communication Parlée”-Stendhal University, Grenoble, France, 1997. Associate Professor at the Faculty of Informatics \& Communication Engineering, at the Arab International University (AIU), Daraa, Syria. She is also a Lecturer, at HIAST, the Syrian Virtual University and the Information Technology Engineering Faculty at Damascus University. Professor Ghneim’s research interests include Artificial Intelligence and Natural Language Processing. She has many publications in Conferences, workshops, journals and books, mainly focusing on Arabic language processing at different modalities (speech, text and image), such as Text-to-Speech, Image Captioning, Sentiment Analysis, Inference detection, … Ghaida Rebdawi. received the DEA and Ph.D. degrees in applied automatic and informatics from the Institut National des Sciences Appliquées de Lyon, France, in 1987 and 1990, respectively. In 1991, she joined the Informatics Department, at the Higher Institute for Applied Sciences and Technology, as a Research Assistant Professor. She is actually a Full Professor and a Research Director, at the forementioned institute. She coauthored many academic books in computer science, and more than 20 articles. Her research interests include software engineering, requirements engineering, and natural language processing. Figure2: E-CONAN Datasets Construction Pipeline mDeBERTa -V3-Base-XNLI-Multilingual-NLI-2mil7 Facebook Bart Large MNLI Ernie-M-Base-MNLI-XNLI Text Deberta-V3-Base-MNL mDeBERTa-V3-Base-MNLI-XNLI Hypothesis Multilingual-MiniLMv2-L12-MNLI-XNLI MiniLM-L6-MNLI FacebookAI RoBERTa-Large-MNLI Multilingual-MiniLMv2-L6-MNLI-XNLI Figure6: Proposed Evaluation of Cross-lingual Baseline Models Text MARBERT Hypothesis Figure7: Proposed Evaluation of Arabic Baseline Model Logical Relation Logical Relation Text Hypothesis Command R7B Arabic Figure8: Proposed Evaluation of LLMs FacebookAI/roberta-large-mnli MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 MoritzLaurer/mDeBERTa-v3-base-mnli-xnli MoritzLaurer/DeBERTa-v3-base-mnli MoritzLaurer/ernie-m-base-mnli-xnli ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN 2-WAY DATASET 0.84 0.83 0.83 0.81 0.77 0.74 0.71 0.8 0.66 0.61 0.55 0.54 0.6 0.6 0.47 0.5 PRECISION ACCURACY ZERO-SHOT CLASSIFICATION RESULTS ON AR ENTAIL DATASET 0.76 0.73 0.72 0.69 0.68 0.67 0.67 0.64 0.59 0.58 0.56 0.56 0.55 0.55 0.53 0.6 ACCURACY PRECISION Figure9: Zero-Shot Classification Results on ArEntail, E-CONAN-2 Datasets ALLAM Qwen2.5 Logical Relation Gemma3 facebook/bart-large-mnli MoritzLaurer/MiniLM-L6-mnli MoritzLaurer/multilingual-MiniLMv2-L6-mnli-xnli MoritzLaurer/multilingual-MiniLMv2-L12-mnli-xnli 0.81 0.79 0.79 0.78 0.78 0.76 0.75 0.74 0.73 0.72 0.8 0.8 0.62 0.59 0.58 0.58 0.55 0.54 0.47 0.5 RECALL F1 0.72 0.72 0.71 0.69 0.68 0.67 0.67 0.66 0.65 0.64 0.61 0.7 0.56 0.55 0.55 0.53 0.53 0.51 0.44 0.5 RECALL F1 FacebookAI/roberta-large-mnli MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 MoritzLaurer/mDeBERTa-v3-base-mnli-xnli MoritzLaurer/DeBERTa-v3-base-mnli MoritzLaurer/ernie-m-base-mnli-xnli ZERO-SHOT CLASSIFICATION RESULTS ON E-CONAN-3 DATASETS 0.72 0.71 0.68 0.64 0.7 0.7 0.61 0.61 0.47 0.46 0.38 0.36 0.36 0.34 0.33 0.3 PRECISION ACCURACY ZERO-SHOT CLASSIFICATION RESULTS ON XNLI DATASET 0.87 0.87 0.87 0.86 0.85 0.81 0.77 0.77 0.57 0.57 0.45 0.42 0.38 0.36 0.35 0.4 PRECISION ACCURACY ZERO-SHOT CLASSIFICATION RESULTS ON ARNLI DATASET 0.55 0.55 0.49 0.48 0.47 0.46 0.5 0.44 0.39 0.37 0.34 0.26 0.25 0.23 0.17 0.15 ACCURACY PRECISION ZERO-SHOT CLASSIFICATION RESULTS ON TEST XNLI DATASET 0.80 0.80 0.80 0.79 0.78 0.73 0.69 0.69 0.57 0.57 0.45 0.45 0.43 0.38 0.37 0.35 ACCURACY PRECISION Figure10: Results on E-CONAN-3, ArNLI and XNLI Datasets facebook/bart-large-mnli MoritzLaurer/MiniLM-L6-mnli MoritzLaurer/multilingual-MiniLMv2-L6-mnli-xnli MoritzLaurer/multilingual-MiniLMv2-L12-mnli-xnli 0.71 0.71 0.68 0.67 0.64 0.64 0.64 0.7 0.7 0.61 0.61 0.46 0.44 0.36 0.34 0.34 0.32 0.27 0.25 F1 RECALL 0.87 0.87 0.86 0.86 0.85 0.85 0.81 0.81 0.81 0.77 0.77 0.57 0.57 0.38 0.36 0.35 0.31 0.27 0.3 RECALL F1 0.59 0.58 0.57 0.54 0.54 0.53 0.50 0.49 0.47 0.46 0.46 0.45 0.36 0.32 0.31 0.28 0.27 0.19 0.2 RECALL F1 0.80 0.80 0.79 0.79 0.78 0.78 0.73 0.73 0.73 0.69 0.69 0.57 0.57 0.38 0.37 0.35 0.31 0.31 0.27 F1 RECALL L L M R E S U L T S O N E- C O N A N - 3 A S 2 - W A Y Cmd_rb Allam 0.96 0.95 0.94 0.92 0.92 0.87 0.81 0.66 0.5 ACCURACY PRECISION Figure11: LLM Zero-Shot Classification on E-CONAN-3 as 2-way LLM RESULTS ON ANS-STANCE cmd_rb Allam 0.91 0.86 0.79 0.73 0.71 0.68 0.67 0.64 0.49 ACCURACY PRECION Figure12: LLM Zero-Shot Classification on Ans-Stance Dataset Qwen2.5 Gemma3 Deepseek 0.95 0.94 0.94 0.93 0.92 0.92 0.91 0.84 0.9 0.33 0.4 RECALL F1 Qwen2.5 Gemma3 Deepseek 0.81 0.71 0.69 0.67 0.64 0.56 0.6 0.49 0.48 0.48 0.42 RECALL F1 LLM RESULTS ON E-CONAN-3 Cmd_rb Allam 0.68 0.68 0.64 0.64 0.61 0.61 0.55 0.48 0.41 ACCURACY PRECISION LLM RESULTS ON ARNLI Cmd_rb Allam 0.68 0.68 0.64 0.62 0.62 0.61 0.61 0.56 0.42 ACCURACY PRECISION LLM RESULTS ON ARSNLI Cmd_rb Allam 0.48 0.48 0.42 0.41 0.38 0.4 0.4 0.32 0.24 ACCURACY PRECISION LLM RESULTS ON ARXNLI Cmd_rb Allam 0.68 0.68 0.64 0.62 0.62 0.61 0.61 0.56 0.42 ACCURACY PRECISION Figure13: LLM Zero-Shot Classification on E-CONAN-3 and SoTA Benchmarks Qwen2.5 Gemma3 Deepseek 0.67 0.67 0.66 0.61 0.48 0.48 0.47 0.44 0.42 0.39 0.38 RECALL F1 Qwen2.5 Gemma3 Deepseek 0.69 0.67 0.66 0.62 0.61 0.61 0.46 0.44 0.43 0.38 0.38 RECALL F1 Qwen2.5 Gemma3 Deepseek 0.58 0.52 0.49 0.49 0.45 0.39 0.38 0.37 0.35 0.32 0.26 RECALL F1 Qwen2.5 Gemma3 Deepseek 0.69 0.67 0.66 0.62 0.61 0.61 0.46 0.44 0.43 0.38 0.38 RECALL F1 TABLE 12 C LASSIFICATION R EPORT OF E-CONAN AND S O TA B ENCHMARKS . Model Dataset Name MarBERT-XNLI-Tuned ArNLI mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 Gemma3 MarBERT-XNLI-Tuned ArXNLI mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 Gemma3 MarBERT-XNLI-Tuned ArSNLI mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 Gemma3 MarBERT-XNLI-Tuned AnsStance mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 Gemma3 MarBERT-XNLI-Tuned E-CONAN-3 mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 Gemma3 F1 Recall Precision Contradiction Contradiction Contradiction Entailment Entailment Entailment Neutral Neutral Neutral 0.28 0.56 0.63 0.78 0.66 0.20 0.41 0.61 0.31 0.38 0.63 0.63 0.75 0.52 0.50 0.51 0.57 0.56 0.55 0.79 0.77 0.77 0.71 0.62 0.70 0.73 0.42 0.63 0.84 0.78 0.62 0.76 0.54 0.69 0.77 0.69 0.91 0.91 0.78 0.88 0.81 0.89 0.90 0.86 0.83 0.84 0.73 0.47 0.50 0.58 0.76 0.63 0.65 0.58 0.51 0.43 0.33 0.50 0.32 0.42 0.50 0.37 0.37 0.80 0.27 0.34 0.32 0.05 0.82 0.4 0.09 0.48 0.32 0.45 0.32 0.05 0.05 0.89 0.09 0.09 0.48 0.96 0.75 0.14 0.76 0.85 0.71 0.85 0.80 0.24 0.96 0.87 0.08 0.76 0.82 0.66 0.85 0.84 0.15 0.90 0.95 0.07 0.59 0.78 0.71 0.86 0.14 0.62 0.67 0.55 0.73 0.66 0.45 0.67 0.67 0.50 0.75 0.80 0.6 0.78 0.67 0.68 0.76 0.73 0.63 0.76 0.79 0.50 0.54 0.53 0.81 0.63 0.64 0.62 Figure15: Confusion Matrix of E-CONAN and SoTA Benchmarks ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral-benchmarks/