Elamur Research⌕
☰
home / papers / 2023 / Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training

Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training

Summary

Arabic has few labelled resources for Natural Language Inference (NLI), the task of judging whether a hypothesis is entailed by, neutral to, or contradicted by a premise. It has even fewer for the related task of Contradiction Detection (CD). The authors address this by assembling a unified Arabic NLI corpus from three public sources: the Arabic part of XNLI, a manually translated Arabic subset of SNLI, and arNLI. They normalise encodings, column names and label schemes before merging, and they keep an English translation alongside every pair.

They then fine-tune two transformers, the Arabic-specific AraBERT and the multilingual XLM-RoBERTa, on both NLI and CD. Before the inference fine-tuning, they add an intermediate step in which the model is further trained on Named Entity Recognition (NER) using ANERcorp.

The main finding is that this NER step raises performance, and it helps the smaller AraBERT more than XLM-RoBERTa. As a result, the language-specific model becomes competitive with the much larger multilingual one. The authors present this as the first large-scale Arabic NLI/CD evaluation and the first use of multi-task pre-training for it. The corpus is published on GitHub.

Key points

  • The merged corpus has 14,758 premise–hypothesis pairs, each with an English translation and a label (0 = entailment, 1 = neutral, 2 = contradiction). It was shuffled and split 60/20/20 into train, test and validation sets.
  • The sources are:
    • the Arabic portion of XNLI;
    • 1,332 hand-translated SNLI pairs from Agić & Schluter;
    • arNLI, with 6,366 items machine-translated from SICK and PHEME.
  • The NER pre-training data is ANERcorp from the CamelLabSplits release: 3,973 Arabic samples, four entity types (person, location, organization, miscellaneous), annotated in the IOB scheme.
  • The models compared are AraBERT and XLM-RoBERTa, evaluated by accuracy and macro-averaged F1. The NER step gives a larger gain to AraBERT, narrowing the gap caused by XLM-RoBERTa's size and training data.
  • Earlier Arabic work on arNLI used hand-crafted features with classical classifiers, where Random Forest reached about 75% accuracy. Planned extensions include POS tagging, Word Sense Disambiguation and Semantic Role Labeling as further pre-training objectives, and testing LLMs such as GPT-4 by recasting classification as generation.

Interesting ideas

  • I1. Inserting a cheap, linguistically grounded auxiliary task (here NER) between generic pre-training and the target inference task can compensate for a smaller model or less data, because it pushes the encoder toward structure the downstream task needs. — For Mova (learn): Before training gparse on silver UD trees, we could run a short intermediate phase on entity or morphology tagging and measure whether attachment accuracy improves on low-data dialect conversions.
  • I2. A compact language-specific model enriched with explicit linguistic signals can rival a large multilingual one, which argues for injecting knowledge rather than scaling. — For Mova (tree): This supports feeding gparse explicit features such as lemmas, morphology and entity spans from the en crate instead of relying only on a bigger encoder.
  • I3. Keeping a parallel English rendering next to every non-English training item makes it easy to inspect errors and compare models across sources. — For Mova (back): When translating through the meta-world, we could store the source text, its graph and the regenerated English together, so each translation failure can be traced to a specific step.
  • I4. The related work recasts reading-comprehension questions as NLI, turning each answer option into a hypothesis checked against the passage; the resulting statements read more naturally than question-plus-option pairs. — For Mova (answer): School-test and fable QA readers could rewrite each answer option as a claim and verify it against the world of the text, giving an explainable entail/contradict verdict per option.
  • I5. The cited contradiction work identifies contradictions through antonymy, polarity flips and numeric mismatch on top of a syntactic parse. — For Mova (world): These three cues map directly onto checks over entity states, negation and measures in the world model, so contradiction detection can report which specific fact clashes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.14666 · arXiv 2307.14666

@misc{deen2023improving,
  title = {{Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training}},
  author = {Deen, Mohammad Majd Saad Al and Pielka, Maren and Hees, Jorn and Abdou, Bouthaina Soulef and Sifa, Rafet},
  year = {2023},
  doi = {10.48550/arxiv.2307.14666},
  eprint = {2307.14666},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.14666}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. MacCartney, Bill and Manning, Christopher: Natural Logic for Textual Inference, 2007. doi : 10.3115/1654536.1654575. ↗
  2. M.-C. de Marneffe, A. N. Rafferty und C. D. Manning. Finding Contradictions in Text. In: Proceedings of ACL-08: HLT. Columbus, Ohio: Association for Computational Linguistics, 2008. url: https://aclantho logy.org/P08-1118. ↗ our card
  3. A. Mishra, D. Patel, A. Vijayakumar, X. Li, P. Kapanipathi und K. Talama- dupula. Reading Comprehension as Natural Language Inference: A Semantic Analysis. In: Proceedings of the Ninth Joint Conference on Lexical and Compu- tational Semantics. Barcelona, Spain (Online): Association for Computational Linguistics, 2020. url: https://aclanthology.org/2020.starsem-1.2. ↗
  4. G. Lai, Q. Xie, H. Liu, Y. Yang und E. Hovy. RACE: Large-scale Reading Comprehension Dataset From Examinations. In: Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, 2017. url: https://aclanthology.org/D17-1082. ↗ our card
  5. K. A. Jallad und N. Ghneim. ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection. In: arXiv preprint arXiv:2209.13953, 2022. ↗ our card
  6. K. Sparck Jones: A statistical interpretation of term specificity and its app- lication in retrieval. en. In: Journal of documentation 28.1., 1972. find
  7. T. Mikolov, K. Chen, G. Corrado und J. Dean: Efficient estimation of word representations in vector space. In: arXiv preprint arXiv:1301.3781, 2013. ↗
  8. L. Pucknat, M. Pielka und R. Sifa. Detecting Contradictions in German Text: A Comparative Study. In: 2021 IEEE Symposium Series on Computational Intelligence (SSCI), 2021. doi: 10.1109/SSCI50451.2021.9659881. ↗
  9. S. R. Bowman, G. Angeli, C. Potts und C. D. Manning. A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics, 2015. url: https://aclanthology.org/D15-1075. ↗
  10. A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk und V. Stoyanov. XNLI: Evaluating Cross-lingual Sentence Representations. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing. Brussels, Belgium: Association for Computational Lingui- stics, 2018. url: https://aclanthology.org/D18-1269. ↗
  11. Ž. Agić und N. Schluter: Baselines and Test Data for Cross-Lingual Infe- rence. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018. find
  12. O. Obeid, N. Zalmout, S. Khalifa, D. Taji, M. Oudah, B. Alhafni, G. Inoue, F. Eryani, A. Erdmann und N. Habash. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. Marseille, Fran- ce: European Language Resources Association, 2020. find our card
  13. W. Antoun, F. Baly und H. Hajj: AraBERT: Transformer-based Model for Arabic Language Understanding, 2020. url: https://arxiv.org/abs/2003.00. ↗ our card
  14. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer und V. Stoyanov: Unsupervised Cross-lingual Representation Learning at Scale, 2019. url: https://arxiv.org/abs/1911.02 ↗
  15. J. Devlin, M.-W. Chang, K. Lee und K. Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: arXiv pre- print arXiv:1810.04805, 2018. ↗
  16. M. Pielka, S. Schmidt, L. Pucknat und R. Sifa. Towards Linguistically Informed Multi-Objective Pre-Training for Natural Language Inference, 2022. arXiv: 22 12.07428 [cs.CL]. find
  17. T. Akiba, S. Sano, T. Yanase, T. Ohta and M. Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. arXiv: 1907.10902 [cs.CL], 2019. ↗ our card
  18. I. Loshchilov and F. Hutter: Decoupled Weight Regularization. In: Proceedings of ICLR 2019, 2019. find
  19. OpenAI: GPT-4 Technical Report. arXiv: 2303.08774, 2023. This figure "fig1.png" is available in "png" format from: http://arxiv.org/ps/2307.14666v1 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deen-2023-improving-natural-language-inference-arabic/