📄 A Dataset for Arabic Textual Entailment

Summary

When this paper was written, Arabic had no dataset for textual entailment (TE). Building one by hand is slow, and the author argues it would also carry the developer's biases. The paper describes a semi-automatic pipeline instead. Headlines from Arabic news sites are sent to Google as queries. Each headline is then paired with an opening paragraph from a different outlet's article on the same story, so the two texts are related but not near-copies. Simple overlap filters are applied. Long Arabic sentences are cut at their first clause-level conjunction, found with MSTParser. Volunteers labelled the pairs in a web tool, with three judgments per pair. Annotator reliability was checked with Fleiss's kappa. The result is ArbTEDS, a first Arabic TE resource. Positive pairs dominate, and requiring unanimous agreement changes the class balance noticeably. For test data, the author proposes taking a non-lead paragraph with high word overlap, which yields more non-entailing pairs. The work matters as a low-cost recipe that can be reused for languages with few resources and only volunteer annotators.

Key points

• ArbTEDS started with 618 T-H pairs drawn at random from thousands collected via Al Jazeera, Al Arabiya and BBC Arabic headlines. After removing 18 pairs marked UN or with no majority, 600 binary pairs remain.
• Pair filters:
  • The headline must be longer than five words.
  • Shared words (surface or lemma) must cover under 80% of the headline length.
  • Both filters were applied to 85% of pairs. The remaining 15% got only the length filter, deliberately keeping some highly similar pairs.
• With a majority label (at least 2 of 3 agree), the data is 478 YES (80%) and 122 NO (20%). With unanimity it is 409 YES (68%) and 69 NO (12%).
• Eight native-speaker PhD volunteers did the annotation. Average T length is about 25 words, H about 10, with roughly 4 words in common.
• Per-annotator Fleiss's kappa ranged from 0.47 to 0.65 and averaged 0.57, the same as the co-annotator-only average. The author concludes that all annotators are acceptably reliable.

Interesting ideas

• I1. Matching a headline from one news source with the opening of another source's story on the same event gives pairs that are related but phrased independently, avoiding near-identical texts.
• I2. Some high-overlap pairs, especially non-entailing ones, are kept on purpose so that systems relying on word overlap fail.
• I3. Long run-on sentences are segmented at the first conjunction that joins two clauses, using a dependency parse. Splitting at the first "and" would often cut through an NP coordination instead.
• I4. Each annotator's reliability is judged by computing kappa with that person included and again with only their co-annotators, then comparing the two values.
• I5. For test data, the non-lead paragraph sharing the most words with the headline is chosen, producing topically close pairs that more often do not entail.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Cross-source reports of the same event could serve as a cheap source of paraphrase and entailment pairs for testing whether two texts produce compatible worlds.
• I2 (learn): Evaluation sets could include a fixed share of lexically similar but semantically different items, as a negative control showing that a gain does not come from bag-of-words shortcuts.
• I3 (tree): Clause-level coordination in the UD tree (conj between verbal heads) could guide splitting long sentences into separate events before frame extraction.
• I4 (learn): The same leave-one-out agreement test could rate each witness parser in silver-data voting and down-weight witnesses whose presence lowers agreement.
• I5 (answer): Distractors for reading-comprehension tests could be built from passages that share the most words with the right answer but do not support it.