📄 Arabic Textual Entailment with Word Embeddings

Summary

The paper addresses Recognizing Textual Entailment (RTE) for Arabic: deciding whether a Hypothesis (H) follows from a Text (T). Arabic is hard to process because of its rich morphology, flexible word order and diglossia, and it lacks large curated resources comparable to the English WordNet or VerbOcean. The authors therefore avoid hand-built lexical resources. They treat entailment as supervised binary classification over two feature groups. The first group is conventional: directional length/overlap counts and Jaccard/Dice similarity at token, lemma and stem levels, plus overlap of named entities extracted with NERAr. The second group uses word2vec embeddings trained on Arabic Gigaword, the ATB and Wikipedia. Each side of a pair becomes an IDF-weighted sum of word vectors, and the authors compute cosine distances between the full texts and between the words left unmatched on each side. With Logistic Regression on the 600-pair ArbTE set and 10-fold cross-validation, the combined system reaches 76.2% accuracy, compared with 69.3% for the best earlier system. This shows that corpus-derived embeddings can partly replace missing lexical resources.

Key points

• Data: ArbTE, 600 manually annotated T–H pairs in MSA drawn from news sites. Embeddings cover about 556K words, using skip-gram, 300 dimensions and window 10, with IDF scores used as input weights. Only about 30 OOV words remain, mostly NEs.
• Preprocessing: SPLIT tags numbers, dates, URLs and punctuation, which are then normalized or removed. Alef/Yaa are normalized. MADAMIRA with the D3 scheme handles tokenization, lemmatization and stemming. Stop words are removed, but negation words are kept.
• Classifiers tested in WEKA: LIBSVM, LIBLINEAR LR and Random Forest. They performed similarly, and LR was slightly best, so only LR results are reported.
• DEV10 accuracy:
  • LR-ALL 76.2
  • LR-TYP 74.33
  • LR-WE 64.67
  • BOW2 baseline (classifier-learned overlap threshold) 72.96
  • SANATE 69.3
  • ETED+ABC 67.3
• Word overlap is highly predictive on this dataset. Embedding features add roughly 2 points on top of the conventional features.

Interesting ideas

• I1. Build a separate "residual" representation from only the words in T that are absent from H, and the reverse, then measure how semantically far those leftovers are. This isolates the part of a pair that overlap measures cannot explain.
• I2. Keep negation words even when other stop words are stripped, because removing them silently reverses entailment decisions.
• I3. Compute the same overlap features at several granularities (surface token, lemma, stem) so that inflectional variation does not hide matches.
• I4. Report a strong overlap baseline whose threshold is learned from the training data. On this dataset it came close to the full systems, which exposes how much of the dataset can be solved by shallow lexical overlap.
• I5. Score named-entity overlap as the fraction of entity words shared, so a partial mention such as "United States" versus "The United States of America" still earns graded credit instead of a yes/no match.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): When checking whether a text supports an answer, examine only the unaligned frame elements of the hypothesis and test whether each is licensed by something in the text, rather than scoring the whole pair at once.
• I2 (tokens): Make sure lemma and stop-word normalization never discards polarity markers, and attach negation explicitly to the token so it reaches the tree and frames.
• I3 (world): When aligning entities between a question and the text world, try matching at several levels (surface form, lemma, WordNet synset) and record which level produced the match, to keep the alignment explainable.
• I4 (learn): Add a learned word-overlap scorer as a negative control in evaluations, so that structure-based gains are only credited where they beat what lexical overlap alone achieves.
• I5 (frames): Feed coreference with graded partial-name overlap as one piece of explainable evidence when linking short and long mentions of the same entity.