Elamur Research⌕
☰
home / papers / 2023 / Automatic Evaluation and Analysis of Idioms in Neural Machine Translation

Automatic Evaluation and Analysis of Idioms in Neural Machine Translation

Summary

Idiomatic phrases such as "under the weather" have figurative meanings, so NMT systems often render them word by word. Research on this problem has been slowed because there was no automatic way to count such literal mistakes.

The authors introduce LitTER, a metric that needs no hand-made blocklists, unlike Shao et al. (2018). It works in three steps:

  • For each annotated source idiom, it builds candidate error lists from bilingual dictionaries (MUSE).
  • It drops any list whose words appear in the reference translation.
  • It flags a hypothesis if it contains a remaining entry.

LitTER is paired with an alignment-based accuracy measure (APT-Eval), and both are macro-averaged per idiom.

Using en→fr and en→es data from Europarl v7, WMT and IWSLT 2017, the authors extract sentences containing 225 English idioms. They train models under zero-shot, joint and upsampled conditions. Initializing from mBART gives large targeted gains, even when the test idioms never occurred in the parallel training data. Probing shows that randomly initialized models largely ignore the context around an idiom, while pretrained ones respond to it.

The work provides a reusable toolkit for evaluating figurative-language translation.

Key points

  • LitTER metric: it reports the share of translations containing dictionary-derived literal renderings of idiom words. Reference filtering removes a word's whole synonym list to avoid false alarms.
  • Data: a manually collected, public list of 225 English idioms. Sentences containing them are matched and span-annotated with a pattern-matching tool built on Spacy's rule-based engine. For each idiom, half the sentence pairs go to idiom-train and half to idiom-test.
  • Training conditions and metrics:
    • Zero: no test idioms in training.
    • Joint: idiom-train added to the regular data.
    • Upsampling: idiom-train repeated N times.
    • Global evaluation uses WMT newstest and IWSLT'17. Targeted evaluation uses LitTER and APT-Eval (unigram precision, ChrF).
  • Model variants: random initialization, mBART, and noisy mBART fine-tuning (masking or replacing 10% of source or target tokens).
  • Limitations and release: LitTER struggles with compound-rich languages such as German and should be used alongside standard metrics. Code and data are at github.com/amazon-research/idiom-mt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 20 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.267

@inproceedings{baziotis2023automatic,
  title = {{Automatic Evaluation and Analysis of Idioms in Neural Machine Translation}},
  author = {Baziotis, Christos and Mathur, Prashant and Hasler, Eva},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.267},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.267}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations, San Diego, CA, USA. 1 ↗ our card
  2. Christos Baziotis, Ivan Titov, Alexandra Birch, and Barry Haddow. 2021. Exploring unsupervised pre-training objectives for machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2956–2971, Online. Association for Computational Linguistics. 4, 5 find
  3. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics. 3 find our card
  4. Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identifiability in transformers. In International Conference on Learning Representations. 7 ↗
  5. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017. Overview of the iwslt 2017 evaluation campaign. In International Workshop on Spoken Language Translation, pages 2–14. 3 find our card
  6. Mathieu Constant, Gülşen Eryiǧit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Survey: Multiword expression processing: A Survey. Computational Linguistics, 43(4):837–892. 1 find our card
  7. Paul Cook, Jelena Mitrović, Carla Parra Escartín, Ashwini Vaidya, Petya Osenova, Shiva Taslimipoor, and Carlos Ramisch, editors. 2021. Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021). Association for Computational Linguistics, Online. 1 find
  8. Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022a. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4154–4175, Dublin, Ireland. Association for Computational Linguistics. 9 find
  9. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022b. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland. Association for Computational Linguistics. 9 find our card
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. 8 find
  11. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. 14 ↗
  12. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. 4, 14 find
  13. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). 1, 3, 8, 12 find
  14. Pablo Gamallo and Marcos Garcia. 2019. Unsupervised compositional translation of multiword expressions. In Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019), pages 40–48, Florence, Italy. Association for Computational Linguistics. 8 find
  15. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. Association for Computational Linguistics. 2, 6, 8 ↗ our card
  16. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. 3, 12 find
  17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, San Diego, CA, USA. 4 ↗
  18. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. 3 find
  19. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. 1 find
  20. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. 4 find our card
  21. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). 4 find
  22. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations. 2 ↗
  23. Dilin Liu. 2017. Idioms: Description, comprehension, acquisition, and pedagogy. Routledge. 1 find
  24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. 1, 4 ↗
  25. Geoffrey Nunberg, Ivan A. Sag, and Thomas Wasow. 1994. Idioms. In Stephen Everson, editor, Language, pages 491–538. Cambridge University Press. 1 find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. 1, 4 find our card
  27. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. 3, 13 find our card
  28. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. 4 find our card
  29. Matı̄ss Rikters and Ondřej Bojar. 2017. Paying attention to multi-word expressions in Neural Machine Translation. In Proceedings of MT Summit XVI. Main Conference: Research Track , volume 1, pages 86–95. 3, 6, 8 ↗
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics. 8 find
  31. Yutong Shao, Rico Sennrich, Bonnie Webber, and Federico Fancellu. 2018. Evaluating machine translation performance on Chinese idioms with a blacklist method. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). 1, 2, 3, 8, 9 you need. In Proceedings of the Advances in Neural Information Processing Systems, pages 5998–6008, Long Beach, CA, USA. 1 find
  32. Elena Voita, Rico Sennrich, and Ivan Titov. 2021. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1126–1140, Online. Association for Computational Linguistics. 4 find
  33. Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896–4907, Online. Association for Computational Linguistics. 2, 6, 8 find
  34. Andrea Zaninello and Alexandra Birch. 2020. Multiword expression aware neural machine translation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3816–3825, Marseille, France. European Language Resources Association. 1, 3, 8 find
  35. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926–5936. PMLR. 4 ↗ our card
  36. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. 8 find our card
  37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. 1 ↗
  38. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477, Punta Cana, Dominican Republic. Association for Computational Linguistics. 6, 8 find our card
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all *Figure 9: Overview of the process for collecting the idiom parallel data.* *Figure 11: Occurrences of idioms in the en→fr idiom-test sets.* *Figure 10: Overview of the process for collecting the idiom parallel data.* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/baziotis-2023-automatic-evaluation-analysis-idioms-neural/