📄 Translating English Discourse Connectives into Arabic: a Corpus-based Analysis and an Evaluation Metric

Summary

Many frequent English discourse connectives, such as since, while and yet, are ambiguous between senses like temporal, causal, contrastive and concessive. Machine translation into Arabic can therefore choose a target word that expresses the wrong relation. The study focused on seven of these connectives: yet, since, meanwhile, while, although, though and even though. They word-aligned the large UN English/Arabic parallel corpus with GIZA++, using MADA segmentation for Arabic, and projected each English connective onto its Arabic counterpart. The resulting frequency tables were cleaned, extended with known translations, and grouped by sense after inspecting example sentences. This produced sense-classified dictionaries of acceptable Arabic equivalents.

These dictionaries drive ACT (Accuracy of Connective Translation), a metric that scores only whether connectives in MT output match the reference or an acceptable same-sense alternative. Some outcomes are flagged for manual review. Judged against human checks, ACT was about 90% accurate. The authors also report benchmark scores for a Moses baseline. The work matters because whole-sentence metrics like BLEU barely register connective errors.

Key points

• Data: UN EN/AR corpus with about 7.1M sentence pairs, 182M English words and 154M Arabic words. Preprocessing was tokenisation and lowercasing for English, and transliteration plus MADA segmentation for Arabic. Arabic forms are shown in Buckwalter transliteration.
• since maps overwhelmingly to the temporal "mn*" (about 78% of 14,324 aligned cases), and its temporal and causal Arabic lists do not overlap.
• Alignment coverage varied a lot. For although, 34,960 of 38,476 occurrences were aligned, but for yet only 996 of 7,087, so the authors inspected the yet sentences by hand.
• Meta-evaluation used 200 UN sentences with 205 connectives. The ACTa score came out about 5% below the human judgement, while ACTa5 and ACTm were off by only about 0.5%. Adding word alignment had improved accuracy in the earlier English-French version.
• Benchmark: a Moses baseline on 2,100 sentences with 2,206 connectives scored BLEU 0.353 and NIST 7.517. It reached ACTa of about 0.55–0.56 and ACTa5 of about 0.64–0.65 across three ACT configurations: no alignment, GIZA++, and MGIZA++ trained on Europarl. 303 cases were left unchecked in the manual-review category.

Interesting ideas

• I1. Discovering a word's senses through translation: projecting an ambiguous source word onto a parallel corpus and clustering its target translations reveals sense distinctions without a sense-annotated corpus.
• I2. Targeted evaluation instead of global scores: a metric restricted to one phenomenon (here connectives) exposes errors that sentence-level n-gram metrics swamp.
• I3. Equivalence lists grouped by sense: a candidate counts as correct if it matches the reference or any same-sense alternative, and it counts as wrong if it comes from a different-sense list.
• I4. An explicit "uncertain, needs a human" bucket: the authors do not force a verdict on unclear matches; they set those cases aside and measured how many of them were in fact correct.
• I5. Validating the metric itself: the authors ran ACT against human judgements to quantify how far it drifts from reality.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Aligned translations could seed or check sense inventories for English connectives and polysemous words in the compiled knowledge layer, with each sense backed by attested target forms.
• I2 (learn): Build narrow scorers (connective relation correct, coreference link correct) so that regressions on rare but meaning-critical items are visible in evaluation.
• I3 (back): When generating or translating from the meaning graph, check that the chosen connective belongs to the sense class of the discourse relation in the graph, not just that it matches one surface word.
• I4 (learn): Silver-data pipelines can keep a separate undecided class when witness parsers disagree and report its size, rather than silently scoring it as right or wrong.
• I5 (learn): Every automatic scorer Mova uses should get its own small human-checked sample, with the error margin reported next to the score.