Elamur Research⌕
☰
home / papers / 2014 / A Human Judgement Corpus and a Metric for Arabic MT Evaluation

A Human Judgement Corpus and a Metric for Arabic MT Evaluation

Paper · 2014 · Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit et al.

Summary

Automatic MT metrics like BLEU reward only exact word and phrase overlap. For a morphologically rich target language such as Arabic, this unfairly punishes translations that use the right stem but a different inflection or clitic. The authors address two gaps. The first is the lack of a sizeable, reliable set of human judgments for English→Arabic MT. They built one by having native speakers rank outputs from six MT systems over news, climate-change and Wikipedia texts. The second is metric design. They propose AL-BLEU, a BLEU variant that gives graded credit when a hypothesis token fails to match the reference exactly but shares its stem or selected morphological features. The weights for these partial credits are tuned on held-out human rankings. On sentence-level Kendall's τ, AL-BLEU agrees with annotators more closely than plain BLEU and slightly better than METEOR. The corpus and code are released, giving Arabic MT evaluation a shared benchmark and showing that cheap morphological analysis can make n-gram metrics fairer for inflection-heavy languages.

Key points

  • Corpus and systems. The corpus combines three sources:

    • NIST 2005 news: 100 documents, 1056 sentences.
    • MEDAR climate texts: 4 documents, 509 sentences.
    • In-house Arabic translations of 7 English Wikipedia articles: 327 sentences.

    It was translated by six English→Arabic systems: four research phrase-based systems and two commercial ones.

  • Annotation. Ten native Arabic annotators used the Appraise toolkit, WMT-2011-style relative ranking with ties allowed, and five candidates per screen. They saw no reference translation. The effort produced about 22,000 rankings covering 1892 sentences.

  • Agreement. Agreement was checked by giving 10% of tasks to a second judge and re-showing 10% of screens to the same judge. Cohen's κ was 0.57 inter-annotator and 0.62 intra-annotator. The comparable WMT figures are 0.41/0.57 for English→European languages on average and 0.40/0.54 for English→Czech.

  • How AL-BLEU scores tokens.

    • An exact match gets full credit.
    • Otherwise, a weighted credit is summed from a stem match and agreement on POS, gender, number, person and definiteness, with features extracted by MADA.
    • The weights are bounded by constraints and optimized by hill-climbing with 100 random restarts on 600 dev sentences. Testing used 300 sentences, split evenly across the three sub-corpora.

  • Results. Test-set τ was 0.3162 for smoothed BLEU, 0.3426 for METEOR and 0.3521 for AL-BLEU (0.3564 for the lexical variant). On the dev set, AL-BLEU reached 0.3759 against BLEU's 0.3361. The Arabic-tuned METEOR performed worse than standard METEOR.

Interesting ideas

  • I1. Replace binary token matching with a ladder of credit levels (exact form, then same stem, then shared grammatical features), and learn the size of each credit from human preferences instead of fixing it by hand.
  • I2. An n-gram that mixes exact and partial matches earns only the credit of its weakest token, so a single sloppy word cannot ride on its neighbours.
  • I3. Measure annotator reliability two ways: send a slice of items to a second judge, and silently re-show a slice to the same judge to catch self-inconsistency.
  • I4. Judges ranked candidates without seeing the reference, so their verdicts reflect quality rather than surface similarity to one gold answer.
  • I5. The authors tested several orderings of the matching levels empirically, and they report that the language-specific METEOR was a weaker baseline than the generic one. Both show that specialised settings should be checked, not assumed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When scoring text regenerated by the text → graph → text compressor, a lemma-plus-features partial match would separate harmless inflection slips from real content loss.
  • I2 (back): Use the same "weakest link" rule when comparing generated phrases to source text, so one wrong argument word cannot hide inside an otherwise correct clause.
  • I3 (learn): Re-ask LLM teacher tiers a hidden subset of identical items, and use both cross-teacher and self-agreement to decide how much their silver labels can be trusted.
  • I4 (learn): When an LLM grades parser or answer outputs, withholding the gold target avoids judgments that simply reward overlap with it.
  • I5 (learn): Always include the plain, untuned configuration as a control when evaluating a "Mova-specialised" variant of a scorer or parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓Carnegie Mellon University QatarDoha · Qatar48 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 4Research subjects and languages the work is about.Machine translation · 1,105 worksSemitic languages · 3,066 worksEvaluation & benchmarks · 1,341 worksMorphology · 420 works

Cite

Cite as Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit and Kemal Oflazer. 2014. A Human Judgement Corpus and a Metric for Arabic MT Evaluation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1026

@inproceedings{bouamor2014human,
  title = {{A Human Judgement Corpus and a Metric for Arabic MT Evaluation}},
  author = {Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit and Kemal Oflazer},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1026},
  url = {https://doi.org/10.3115/v1/d14-1026}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy.
  2. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland.
  3. Boxing Chen and Roland Kuhn. 2011. AMBER: A Modified BLEU, Enhanced Ranking Metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 71–77, Edinburgh, Scotland. find
  4. Jacob Cohen. 1968. Weighted Kappa: Nominal Scale Agreement Provision for Scaled Disagreement or Partial Credit. Psychological bulletin, 70(4):213. ↗
  5. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011. TESLA at WMT 2011: Translation Evaluation and Tunable Metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 78–84, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  6. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation, Edinburgh, UK.
  7. Etienne Denoual and Yves Lepage. 2005. BLEU in Characters: Towards Automatic MT Evaluation in Languages Without Word Delimiters. In Proceedings of the Second International Joint Conference on Natural Language Processing, Jeju Island, Republic of Korea. find
  8. Ahmed El Kholy and Nizar Habash. 2011. Automatic Error Analysis for Morphologically Rich Languages. In Proceedings of the MT Summit XIII, pages 225–232, Xiamen, China. find
  9. Ahmed El Kholy and Nizar Habash. 2012. Orthographic and Morphological Processing for English-Arabic Statistical Machine Translation. Machine Translation, 26(1):25–45. ↗
  10. Christian Federmann. 2012. Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output. The Prague Bulletin of Mathematical Linguistics, 98(1):25–35. ↗
  11. N. Habash, O. Rambow, and R. Roth. 2009. Mada+ Tokan: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt. find
  12. Petr Homola, Vladislav Kuboň, and Pavel Pecina. 2009. A Simple Automatic MT Evaluation Metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 33–36, Athens, Greece, March. Association for Computational Linguistics. find
  13. Maurice G Kendall. 1938. A New Measure of Rank Correlation. Biometrika. ↗
  14. J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159–174. ↗
  15. Ding Liu and Daniel Gildea. 2005. Syntactic Features for Evaluation of Machine Translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32.
  16. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2010. TESLA: Translation Evaluation of Sentences with Linear-Programming-Based Analysis. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics (MATR), pages 354–359. find
  17. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. find
  18. Bente Maegaard, Mohamed Attia, Khalid Choukri, Olivier Hamon, Steven Krauwer, and Mustafa Yaseen. 2010. Cooperation for Arabic Language Resources and Tools–The MEDAR Project. In Proceedings of LREC, Valetta, Malta.
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania.
  20. Maja Popović and Hermann Ney. 2009. Syntax-oriented Evaluation Measures for Machine Translation Output. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 29– 32, Athens, Greece. find
  21. Maja Popović. 2011. Morphemes and POS Tags for n-gram Based Evaluation Metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 104–107, Edinburgh, Scotland. find
  22. Stuart Russell and Peter Norvig. 2009. Artificial Intelligence: A Modern Approach. Prentice Hall Englewood Cliffs. find
  23. Matthew Snover, Bonnie J. Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of AMTA, Boston, USA.
  24. Matthew Snover, Nitin Madnani, Bonnie J. Dorr, and Richard Schwartz. 2010. TER-Plus: Paraphrase, Semantic, and Alignment Enhancements to Translation Edit Rate. Machine Translation, 23(2-3). ↗
  25. Radu Soricut and Eric Brill. 2004. A Unified Framework For Automatic Evaluation Using 4-Gram Co-occurrence Statistics. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 613–620, Barcelona, Spain, July. find
  26. Cüneyd Tantug, Kemal Oflazer, and Ilknur Durgar El-Kahlout. 2008. BLEU+: a Tool for Fine-Grained BLEU Computation. In Proceedings of the 6th edition of the Language Resources and Evaluation Conference, Marrakech, Morocco. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bouamor-2014-human-judgement-corpus-metric-arabic/