Elamur Research⌕
☰
home / papers / 2016 / Machine Translation Evaluation for Arabic using Morphologically-enriched Embeddings

Machine Translation Evaluation for Arabic using Morphologically-enriched Embeddings

Paper · 2016 · Francisco Guzmán, Houda Bouamor, Ramy Baly et al.

Summary

Automatic metrics for translation into morphologically rich languages handle three cases poorly: they cannot reliably separate wrong word choices, acceptable rephrasings, and inflection errors such as broken agreement or wrong case. The authors address this for Arabic. They adapt a feed-forward pairwise neural network, originally built for English MT evaluation, that decides which of two hypotheses better matches a reference. The inputs are sentence embeddings built at several levels of analysis. Lexical levels are surface tokens, normalized tokens and lemmas, segmented with MADAMIRA under the ATB scheme. Morpho-syntactic levels are sequences of POS tags from four tagsets of very different granularity. Cosine similarities and n-gram metrics enter the network as extra features. Combining lexical and rich morpho-syntactic embeddings raised correlation with human pairwise judgments by roughly 75% over the previous state of the art. The finest tagset, Buckwalter POS, was the strongest single representation. This matters because metrics are used to tune systems as well as score them. A metric that notices agreement errors can steer translation into rich-morphology languages more effectively.

Key points

  • Architecture: The input is a triple (reference, hypothesis 1, hypothesis 2). Three tanh hidden groups model each pairwise interaction, and a sigmoid output says which hypothesis is better.

    • Skip arcs feed cosine similarities and AL-BLEU, METEOR and NIST scores straight to the output.
    • Training uses SGD with adagrad and L2 regularization, implemented in Theano.

  • Scoring a single translation: Each translation is compared against an "average" translation built from the training data, running the network in both directions because it is not symmetric. The result is rescaled to [0, 1].

  • Tagsets: CATiBPOS has 6 base tags, Stanford POS 32 (via Stanford CoreNLP), Kulick POS 43 and Buckwalter POS 485. The first three come from MADAMIRA.

  • Single-representation results (Kendall's τ on TEST):

    RepresentationKendall's τ
    Buckwalter POS25.49
    TOKEN24.35
    NORM23.22
    LEMMA21.17
    Kulick16.25
    Stanford10.90
    CATiB5.41

    The best combination was Buckwalter POS + TOKEN.

  • Expressiveness check: The spread (standard deviation) of cosine similarity scores across a representation correlates with its τ at ρ = 0.94. Lexical representations lose some expressiveness because of OOV tokens (about 1.3%).

Interesting ideas

  • I1. Instead of producing an absolute score, train a judge to pick the better of two candidates against a reference, then obtain an absolute score by comparing each candidate with an averaged "typical" candidate in both directions.
  • I2. A sentence made only of very fine morphological tags, with no words at all, carries enough signal to detect agreement errors in gender and number. It was the best single feature, which shows that rich tags are a meaningful view of a sentence on their own.
  • I3. Using one coarse tag for every negative particle let the model penalize a hypothesis that wrongly introduced negation more heavily than one that merely chose a different word. Abstracting closed-class words into function categories exposes meaning shifts that surface tokens hide.
  • I4. Before training anything, a representation's usefulness can be estimated from how widely its similarity scores vary across a test set. Representations that yield nearly constant similarities carry little information.
  • I5. Lexical and morpho-syntactic similarities capture complementary aspects, roughly meaning versus grammatical form, and combining them beat both n-gram metrics and either view alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness-parser disagreements could be resolved by a pairwise judge that ranks two candidate trees or answers, with the margin against an average candidate serving as a calibrated confidence for silver data.
  • I2 (tokens): The morphology output of crate en could be exported as a tag-sequence fingerprint and used to check number and person agreement between subject and verb before parsing.
  • I3 (frames): Polarity and modality markers (not, never, will, might) could be mapped to shared categories so frame comparison immediately flags a flipped negation instead of treating it as a lexical difference.
  • I4 (learn): This is a cheap screening test for candidate features or embedding levels, run alongside negative controls to discard uninformative views early.
  • I5 (back): The text → graph → text compressor could be evaluated by comparing regenerated text with the original at both the lemma level and the morphological-tag level, catching paraphrase drift and inflection errors separately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksMorphology · 405 works

Cite

Cite as Francisco Guzmán, Houda Bouamor, Ramy Baly and Nizar Habash. 2016. Machine Translation Evaluation for Arabic using Morphologically-enriched Embeddings. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{guzman2016machine,
  title = {{Machine Translation Evaluation for Arabic using Morphologically-enriched Embeddings}},
  author = {Francisco Guzmán, Houda Bouamor, Ramy Baly and Nizar Habash},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1132/}
}

References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473. ↗
  2. Yoshua Bengio and Xavier Glorot. 2010. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In Proceedings of AISTATS 2010, volume 9, pages 249–256. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3(Feb):1137–1155.
  4. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference, SciPy ’10, Austin, Texas. find
  5. Jan A Botha and Phil Blunsom. 2014. Compositional Morphology for Word Representations and Language Modeling. In ICML, pages 1899–1907. find
  6. Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit, and Kemal Oflazer. 2014. A Human Judgement Corpus and a Metric for Arabic MT Evaluation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 207–213, Doha, Qatar.
  7. Tim Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. Linguistic Data Consortium, University of Pennsylvania, 2002. LDC Catalog No.: LDC2004l02. Technical report, ISBN 1-58563-324-0. ↗
  8. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy.
  9. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland.
  10. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada.
  11. Boxing Chen and Roland Kuhn. 2011. AMBER: A Modified BLEU, Enhanced Ranking Metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 71–77, Edinburgh, Scotland. find
  12. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN encoder-decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078. ↗
  13. Alexander Clark. 2003. Combining Distributional and Morphological Information for Part of Speech Induction. In Proceedings of the Tenth Conference on European chapter of the Association for Computational Linguistics, pages 59–66.
  14. Ryan Cotterell and Hinrich Schütze. 2015. Morphological Word-Embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1287–1292, Denver, Colorado. find
  15. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011. Tesla at WMT 2011: Translation Evaluation and Tunable Metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 78–84. find
  16. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation, Edinburgh, UK.
  17. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1370–1380, Baltimore, Maryland.
  18. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality using n-gram Co-occurrence Statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138–145. Morgan Kaufmann Publishers Inc. ↗
  19. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12:2121–2159. find
  20. Jesús Giménez and Lluís Màrquez. 2007. Linguistic Features for Automatic Evaluation of Heterogenous MT Systems. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 256–264. find
  21. Yoav Goldberg. 2015. abs/1510.00726. A Primer on Neural Network Models for Natural Language Processing. ↗
  22. CoRR, Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015. ReVal: A Simple and Effective Machine Translation Evaluation Metric Based on Recurrent Neural Networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1066–1072, Lisbon, Portugal. find
  23. Francisco Guzmán, Shafiq Joty, Lluís Màrquez, Alessandro Moschitti, Preslav Nakov, and Massimo Nicosia. 2014a. Learning to differentiate better from worse translations. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 214–220, Doha, Qatar, October. Association for Computational Linguistics. find
  24. Francisco Guzmán, Shafiq Joty, Lluís Màrquez, and Preslav Nakov. 2014b. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 687–698, Baltimore, Maryland. find
  25. Francisco Guzmán, Shafiq Joty, Lluís Màrquez, and Preslav Nakov. 2015. Pairwise neural machine translation evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 805–814, Beijing, China. find
  26. Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2016. Machine translation evaluation meets community question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 460–466, Berlin, Germany. find
  27. Nizar Habash and Fatiha Sadat. 2006. Arabic Preprocessing Schemes for Statistical Machine Translation. In Proceedings of the Human Language Technology Conference of the NAACL, pages 49–52. Association for Computational Linguistics.
  28. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  29. Nizar Habash, Reem Faraj, and Ryan Roth. 2009. Syntactic Annotation in the Columbia Arabic Treebank. In Proceedings of MEDAR International Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
  30. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  31. Shafiq Joty, Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2014. DiscoTK: Using Discourse Structure for Machine Translation Evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT ’14, pages 402–408, Baltimore, Maryland, USA. find
  32. Seth Kulick, Ryan Gabbard, and Mitchell Marcus. 2006. Parsing the Arabic Treebank: Analysis and Improvements. In Proceedings of the Treebanks and Linguistic Theories Conference, pages 31–42. Citeseer. find
  33. Ding Liu and Daniel Gildea. 2005. Syntactic Features for Evaluation of Machine Translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32.
  34. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2010. TESLA: Translation Evaluation of Sentences with Linear-Programming-Based Analysis. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics (MATR), pages 354–359. find
  35. Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully Automatic Semantic MT Evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 243–252. find
  36. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In ACL (System Demonstrations), pages 55–60.
  37. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent Neural Network Based Language Model. In Interspeech, volume 2, page 3. find
  38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  39. Jeff Mitchell and Mirella Lapata. 2010. Composition in Distributional Models of Semantics. Cognitive Science, 34(8):1388–1439. ↗
  40. Preslav Nakov, Francisco Guzman, and Stephan Vogel. 2012. Optimizing for Sentence-Level BLEU+1 Yields Short Translations. In Proceedings of COLING 2012, pages 1979–1994, Mumbai, India. find
  41. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In LREC, volume 14, pages 1094–1101.
  42. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas, AMTA ’06, Cambridge, Massachusetts, USA.
  43. Matthew Snover, Nitin Madnani, Bonnie J. Dorr, and Richard Schwartz. 2010. TER-Plus: paraphrase, semantic, and alignment enhancements to Translation Edit Rate. Machine Translation, 23(2-3). ↗
  44. Richard Socher, John Bauer, Christopher D. Manning, and Ng Andrew Y. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 455–465, Sofia, Bulgaria. find
  45. Radu Soricut and Eric Brill. 2004. A Unified Framework For Automatic Evaluation Using 4-Gram Co-occurrence Statistics. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, pages 613–620, Barcelona, Spain. find
  46. Cüneyd Tantug, Kemal Oflazer, and Ilknur Durgar El-Kahlout. 2008. BLEU+: a Tool for Fine-Grained BLEU Computation. In Proceedings of the 6th edition of the Language Resources and Evaluation Conference, Marrakech, Morocco. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guzman-2016-machine-translation-evaluation-arabic-morphologically/