Elamur Research⌕
☰
home / papers / 2016 / Applying the Cognitive Machine Translation Evaluation Approach to Arabic

Applying the Cognitive Machine Translation Evaluation Approach to Arabic

Summary

The paper asks whether a cognitively motivated scheme for judging how hard machine-translated text is to post-edit, first built for Russian, Spanish and Bulgarian, still works for a typologically distant language, Modern Standard Arabic. The scheme comes from Temnikova (2010). It takes error classes derived from Vilar et al. (2006) and orders them by the mental effort a corrector is assumed to need, based on psycholinguistic findings. The goal is fairer pay for post-editors than time, edit distance or BLEU would give. The authors took three English Wikinews articles translated into Arabic with Google Translate and chose them so that their output quality differed. Trained native speakers post-edited them in the QALB framework, and four annotators then assigned an error category to each correction. The category set was kept almost unchanged so the port itself could be tested. Error distributions shifted as MT quality improved. In all four languages, the better translations showed a similar pattern. This suggests that simple error counting with a ranked taxonomy can work across language families without per-language engineering.

Key points

  • Data: three news texts, labelled Ar46, Ar37 and Ar14 after their share of corrected tokens (46%, 37%, 14%).
    • Their MT outputs were 193, 194 and 134 tokens long.
    • They come from the QALB project, which aimed at a 2M-word error-corrected Arabic corpus.
  • Pre-processing: MADAMIRA automatically fixed common Arabic orthographic slips before human post-editing, such as Ya/Alif-Maqsura, Ha/Ta-Marbuta and Hamza forms.
  • Editing tool: post-editing used the QAWI web tool, which logs word-level actions (modify, move, delete, merge, split, insert) to XML.
  • Taxonomy changes: the original ranking runs from in-word fixes (CInF) up to sentence-level problems (InP, MissP, WoW, WoPh). The only change was merging WoW and WoPh into one class, WoE, plus a flag for words that received two edits.
  • Agreement: inter-annotator agreement was 88.49% (Ar46), 88.50% (Ar37) and 99% (Ar14), averaging 92%. Annotation was done in Google Spreadsheets with drop-down category lists and guidelines translated into Arabic.

Interesting ideas

  • I1. Instead of counting errors flatly, weight each error type by how much context a human must hold in mind to fix it: inside a word, one word, or the whole sentence. This gives a transparent cost score that can be compared across systems and languages. — For Mova (learn): parser and frame errors in evaluation reports could be bucketed by repair scope (token-local vs. clause-wide), so a regression on hard, structural errors is not hidden by gains on cheap ones.
  • I2. When porting an evaluation scheme to a new language, the authors kept the category set almost frozen so that any mismatch would show up as a finding rather than be patched away. — For Mova (tree): when adapting UD dialect conversion rules to a new treebank, first run the unchanged rules and log where they fail, then adapt only those spots.
  • I3. Running a deterministic normalizer before human correction removes trivial, predictable errors, so annotators spend effort on substantive problems and the effort signal stays clean. — For Mova (learn): apply rule-based spelling and tokenization normalization before witness parsers vote or the LLM teacher is queried, so disagreements reflect real syntactic ambiguity.
  • I4. Logging corrections as typed operations (move, merge, split, insert) and dividing by token count gives a cheap, explainable quality proxy for a text. — For Mova (back): score text → graph → text round trips by the typed edit operations needed to restore the original, normalized by length, rather than by a single opaque similarity number.
  • I5. A separate flag for "two fixes on the same word" captures compound repairs that a one-label scheme would undercount. — For Mova (learn): self-induced repair rules could mark cases needing chained operations, such as reattach plus relabel, and treat them as a distinct, costlier error class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{temnikova2016applying,
  title = {{Applying the Cognitive Machine Translation Evaluation Approach to Arabic}},
  author = {Temnikova, Irina and Zaghouani, Wajdi and Vogel, Stephan and Habash, Nizar},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/2adqvo3tyyot},
  url = {https://doi.org/10.63317/2adqvo3tyyot}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, J. (2001). Postediting: An integrated part of a translation software program. Language International, 13(2):26–29. find
  2. Avramidis, E., Burchardt, A., Hunsicker, S., Popovic, M., Tscherwinka, C., Vilar, D., and Uszkoreit, H. (2014). The taraXU corpus of human-annotated machine translations. LREC. find
  3. Baddeley, A. D. and Hitch, G. J. (1974). Working memory. The psychology of learning and motivation, 8:47–89. find
  4. Blain, F., Senellart, J., Schwenk, H., Plitt, M., and Roturier, J. (2011). Qualitative analysis of post-editing for high quality machine translation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), pages 164–171. find
  5. Carl, M., Dragsted, B., Elming, J., Hardt, D., and Jakobsen, A. L. (2011). The process of post-editing: A pilot study. In Proceedings of the 8th international NLPSC workshop. Special theme: Human-machine interaction in translation, pages 131–142. find
  6. De Sousa, S. C., Aziz, W., and Specia, L. (2011). Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In RANLP, pages 97–103. find
  7. Doherty, S. and O’Brien, S. (2009). Can MT output be evaluated through eye tracking. MT Summit XII: proceedings of the twelfth Machine Translation Summit, pages 214–221. find
  8. Farrús Cabeceran, M., Ruiz Costa-Jussà, M., Mariño Acebal, J. B., Rodrı́guez Fonollosa, J. A., et al. (2010). Linguistic-based evaluation criteria to identify statistical machine translation errors. find
  9. Fishel, M., Bojar, O., and Popovic, M. (2012). Terra: A collection of translation error-annotated corpora. In LREC, pages 7–14. find
  10. Flanagan, M. (1994). Error classification for MT evaluation. In Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, pages 65–72. find
  11. Font-Llitjós, A., Carbonell, J. G., and Lavie, A. (2005). A framework for interactive and automatic refinement of transfer-based machine translation. Computer Science Department, page 286. find
  12. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. find
  13. Harley, T. A. (2013). The Psychology of Language: From data to theory. Psychology Press. find
  14. Koponen, M., Aziz, W., Ramos, L., and Specia, L. (2012). Post-editing time as a measure of cognitive effort. Proceedings of WPTP. find
  15. Koponen, M. (2012). Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 181–190. Association for Computational Linguistics. find
  16. Krings, H. P. and Koby, G. S. (2001). Repairing texts: Empirical investigations of machine translation post-editing processes, volume 5. Kent State University Press. find
  17. Lacruz, I. and Munoz Martin, R. (2014). Pauses and objective measures of cognitive demand in post-editing. Americal Translation and Interpreting Studies Association Conference. find
  18. Larigauderie, P., Gaonac’h, D., and Lacroix, N. (1998). Working memory and error detection in texts: What are the roles of the central executive and the phonological loop? Applied Cognitive Psychology, 12(5):505–527. find
  19. Mohit, B., Rozovskaya, A., Habash, N., Zaghouani, W., and Obeid, O. (2014). The first QALB Shared Task on automatic text correction for Arabic. ANLP 2014, page 39. find
  20. Obeid, O., Zaghouani, W., Mohit, B., Habash, N., Oflazer, K., and Tomeh, N. (2013). A web-based annotation framework for large-scale text correction. In Sixth International Joint Conference on Natural Language Processing, page 1. find
  21. O’Brien, S. (2011). Towards predicting post-editing productivity. Machine translation, 25(3):197–215. find
  22. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311– 318. Association for Computational Linguistics. find our card
  23. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland. find our card
  24. Rozovskaya, A., Bouamor, H., Habash, N., Zaghouani, W., Obeid, O., and Mohit, B. (2015). The second QALB Shared Task on automatic text correction for Arabic. find
  25. ANLP Workshop 2015, page 26. find
  26. Somers, H. (2003). Computers and translation: A translator’s guide, volume 35. John Benjamins Publishing. find
  27. Specia, L. (2011). Exploiting objective annotations for measuring translation post-editing effort. find
  28. Stymne, S., Danielsson, H., Bremin, S., Hu, H., Karlsson, J., Lillkull, A. P., and Wester, M. (2012). Eye tracking as a tool for machine translation error analysis. In LREC, pages 1121–1126. find
  29. Stymne, S. (2011). Blast: A tool for error analysis of machine translation output. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations, pages 56–61. Association for Computational Linguistics. find our card
  30. Temnikova, I., Orasan, C., and Mitkov, R. (2012). CLCM - A linguistic resource for effective simplification of instructions in the crisis management domain and its evaluations. In LREC, pages 3007–3014. find
  31. Temnikova, I. (2010). Cognitive evaluation approach for a controlled language post–editing experiment. In LREC. find
  32. Vieira, L. N. (2014). Indices of cognitive effort in machine translation post-editing. Machine Translation, 28(3-4):187–216. find
  33. Vilar, D., Xu, J., d’Haro, L. F., and Ney, H. (2006). Error analysis of statistical machine translation output. In Proceedings of LREC, pages 697–702. find our card
  34. Zaghouani, W., Habash, N., and Mohit, B. (2014a). The Qatar Arabic Language Bank guidelines. In Technical Report CMU-CS-QTR-124, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, September. find
  35. Zaghouani, W., Mohit, B., Habash, N., Obeid, O., Tomeh, N., Rozovskaya, A., Farra, N., Alkuhlani, S., and Oflazer, K. (2014b). Large scale Arabic error annotation: Guidelines and framework. In International Conference on Language Resources and Evaluation (LREC 2014). find our card
  36. Zaghouani, W., Habash, N., Bouamor, H., Rozovskaya, A., Mohit, B., Heider, A., and Oflazer, K. (2015). Correction annotation for non-native Arabic texts: Guidelines and corpus. Proceedings of The 9th Linguistic Annotation Workshop, pages 129–139. find
  37. Zaghouani, W., Habash, N., Obeid, O., Mohit, B., and Oflazer, K. (2016). Building an Arabic Machine Translation Post-Edited Corpus: Guidelines and Annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/temnikova-2016-applying-cognitive-machine-translation-evaluation/