Elamur Research⌕
☰
home / papers / 2008 / Exploitation of an Arabic Language Resource for Machine Translation Evaluation: using Buckwalter-based Lookup Tool to Augment CMU Alignment Algorithm

Exploitation of an Arabic Language Resource for Machine Translation Evaluation: using Buckwalter-based Lookup Tool to Augment CMU Alignment Algorithm

Paper · 2008 · Clare Voss, Jamal Laoudi and Jeffrey Micher

Summary

The paper addresses a known weakness in machine translation evaluation. At the level of single segments, human adequacy judgments and automatic metrics agree poorly, as the authors' earlier study of three DARPA GALE Arabic-English systems showed. Their response is to "ground" each MT output against its Reference Translation (RT) before any scoring. The pair is first labelled with the kinds of comparisons a human judge makes intuitively.

The pipeline has three phases:

  1. The word aligner from METEOR (CMU) finds exact, stemmed and WordNet-synonym hits.
  2. The authors' Buckwalter-based Lookup Tool (BBLT) lists candidate English translations for every Arabic source token. This extra evidence sorts the leftover tokens into six further categories, covering both credit and error.
  3. An adequacy score is assigned from the categorised pair. This phase is described but not presented.

The method is shown on a Boeing newswire sentence and on one sentence translated by six Arabic-English engines. The result is colour-coded annotations that expose differences plain metrics miss. The work matters because it gives credit for correct output absent from the reference and gives errors a type, opening a path to metrics closer to human judgment.

Key points

  • The motivation is the weak segment-level correlation between subjective scores and metrics such as BLEU, NIST and METEOR (Voss et al. 2006, DARPA GALE newswire).
  • There are seven colour-coded categories:
    • (i) exact/synonym/stem hits, in green;
    • (ii) RT gaps, in blue;
    • (iii) paraphrase hits, in blue;
    • (iv) RT-MT dual divergences, such as "and" vs "or", in purple;
    • (v) NFW (not-found-word) transliterations of OOV words, such as "Alam", in purple;
    • (vi) MT gaps, in yellow;
    • (vii) MT lexical selection errors, in red;
    • near-miss mistranslations get gray for partial credit.
  • The CMU aligner output is extended into an eight-column table. Exact and synonym matches are auto-accepted. "Artificial" matches, usually unrelated closed-class pairs like "a"/"the", are flagged for human review, which is the only manual step.
  • BBLT gives each source token its own column of candidate translations and has an optional transliteration mode, so readers who do not know Arabic can inspect the alignments.
  • In the six-engine example, the engines receive different category profiles. For instance, MT3 transliterated the verb and MT6 mistranslated it, and MT4 moved a temporal phrase to the front.

Interesting ideas

  • I1. Use a source-side lexicon to widen the reference, so a correct word the human translator left out counts as an RT gap rather than an error.
  • I2. Separate "wrong but dictionary-licensed choice" from "hallucination" by checking whether an output word matches any lexicon translation of any source word.
  • I3. Detect metric gaming by finding reference items that the source lexicon supports but the output silently drops.
  • I4. Automate confident matches and send only one narrow, ambiguous class (closed-class "artificial" alignments) to a human.
  • I5. Replace one adequacy number with a typed error and credit profile per segment, which is easier to explain and to weight later.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): when scoring text regenerated from the meaning graph, accept tokens licensed by graph nodes even if the reference sentence lacks them.
  • I2 (back): in translation through the meta-world, flag output words that no entity or frame in the graph licenses as hallucinations, distinct from mere wording choices.
  • I3 (learn): add a negative-control check that penalises a model which improves a score by omitting hard content words or arguments.
  • I4 (learn): route only low-confidence or function-word disagreements between witness parsers to the LLM teacher tier, keeping supervision cheap.
  • I5 (answer): report reader failures by category, such as missed role, wrong lexical choice or dropped entity, instead of a bare accuracy figure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬DEVCOM Army Research LaboratoryAdelphi · United States14 works · 2,292 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Clare Voss, Jamal Laoudi and Jeffrey Micher. 2008. Exploitation of an Arabic Language Resource for Machine Translation Evaluation: using Buckwalter-based Lookup Tool to Augment CMU Alignment Algorithm. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{voss2008exploitation,
  title = {{Exploitation of an Arabic Language Resource for Machine Translation Evaluation: using Buckwalter-based Lookup Tool to Augment CMU Alignment Algorithm}},
  author = {Clare Voss, Jamal Laoudi and Jeffrey Micher},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/25omz4jjwok5},
  url = {https://doi.org/10.63317/25omz4jjwok5}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Buckwalter Arabic Morphological Analyzer (BAMA), Version 2.0, LDC Catalog number LDC2004L02, www.ldc.upenn.edu/Catalog Ellsworth, M. and A. Janin (2007) “Mutaphrase: Paraphrasing with FrameNet.” In Proceedings of ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic.
  2. Habash, Nizar. (2004) Aragen: Large Scale Arabic Morphological Generation. Poster Presentation. TECH 2004. University of Maryland College Park. March 19, 2004 http://clipdemos.umiacs.umd.edu/Aragen/ ↗
  3. Jayaraman, S.and A. Lavie (2005) "Multi-Engine Machine Translation Guided by Explicit Word Matching". In Proceedings of the 10th Annual Conference of the European Association for Machine Translation (EAMT-2005), Budapest, Hungary. find
  4. Lavie, A., K. Sagae and S. Jayaraman.(2004) "The Significance of Recall in Automatic Metrics for MT Evaluation" In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas, Washington, DC. find
  5. Lavie, A. and A. Agarwal, (2007) "METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments" In Proceedings of the Second Workshop on Statistical Machine Translation at the 45th Meeting of the Association for Computational Linguistics, Prague, Czech Republic.
  6. Strassel, Stephanie and Andrew W. Cole (2006) “Corpus Development and Publication” In Proceedings of LREC, Genoa,Italy .http://papers.ldc.upenn.edu/LREC2006/Co rpusDevelopmentAndPublication.pdf Voss, C., J. Micher, J. Laoudi, C. Tate (2006) “Ongoing Machine Translation Evaluation at ARL,” Presentation, In Proceedings of the NIST Machine Translation Workshop, Washington, DC. ↗
  7. Zawaydeh, B. and Z. Saadi (2006) “Orthographic Variations in Arabic Corpora,” Government Users Conference,www.basistech.com/knowledge-center/Ara bic/orthographic-variations-in-arabic.pdf Table 2. Screenshot of Results Table from Buckwalter-Based Look-up Tool (BBLT) for SL segment in Figure 1. + BBLT Alignment <notes> RT MT CMU Algorithm 1 A 1 The 2 Boeing 2 Boeing 3 737-300 3 737-300 artificial exact exact Review: < indef vs. def article> Accept Accept 4 aircraft BBLT: BBLT: RT gap RT correct 4 requires 5 to 5 a 6 runway 6 included 7 that 8 is 7 the 13 long 8 length 9 of BBLT: MT paraphrase BBLT: 9 at 10 at 10 least 11 least 11 2200 12 2200 12 meters 13 metres exact Accept exact Accept exact Accept wn_synonymy Accept 14 for exact exact Accept Accept 15 take 16 off 16 take 17 off 17 and 15 or BBLT <idiomatic> Accept 18 landing14 landing exact Table 3. Results of CMU alignment (boxed) and Results of combined CMU + BBLT matching analysis with categories CMU+BBLT category MT partial MT correct MT correct vii i. i. MT correct MT gap ii. vi MT error vii iii MT correct MT correct MT correct MT correct i. i i i. MT correct MT correct i. i. MT dual diverg iv MT correct i Source Language Text: ءﺎﻌﺑرﻻا مﻮﻳ ﺔآﺮﻌﻤﻟا ءاﺪﺘﺑ ا ﺦﻳرﺎﺘﺑ ﺶﻴﺠﻟا ﺪﺋﺎﻗ ﻢﻠﻋ . Reference Translation: The Army commander found out about the start date of the battle on Wednesday. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/voss-2008-exploitation-arabic-language-resource-machine/