Elamur Research⌕
☰
home / papers / 2023 / Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting

Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting

Summary

Idioms carry meanings that cannot be assembled from their individual words, and neural MT systems, including commercial ones, still tend to render them literally. The authors define non-compositional translation formally and use that definition to build a synthetic translation task. In it, transformer models stay word-for-word until non-compositional patterns reach a certain share of the training data, and then switch abruptly to the correct translation.

For real languages, they built evaluation sets of idiomatic, literal-usage and random sentences in French, Finnish and Japanese. Sentences were drawn from OpenSubtitles by matching idiom lemmas. They then tried to improve the pretrained ∆LM (DeltaLM) model with two lightweight methods: giving extra loss weight to training sentences that may contain idioms, and adding kNN-MT retrieval. Together these raise accuracy on idiomatic sentences by up to 13 points absolute, without hurting literal-usage sentences. Out-of-domain quality for French and Finnish also improves. The work shows that existing parallel data can be exploited better for long-tail phenomena such as idioms.

Key points

  • The idiom lists come from language-learning sites: 148 French, 92 Finnish and 1336 Japanese idioms. These yielded a dataset of about 4k natural test sentences, split into idiomatic and literal sets plus length-matched random sets (in-domain OpenSubtitles and out-of-domain Ted Talks).
  • To build the data, sentences were matched in OpenSubtitles 2018 using Stanza lemmatization. The 10% of pairs with the lowest COMET-QE scores were discarded as likely misaligned.
  • Native annotators checked the idiomatic and literal labels, with moderate agreement (Krippendorff's α of 0.5754 for French and 0.6454 for Finnish).
  • Upweighting plus kNN-MT improved idiomatic accuracy by 10.4 points absolute on average. Human evaluation found that severe semantic errors dropped by 7.52 points on average.
  • On the Finnish random set, the severe error rate fell from 0.1317 to roughly 0.06. Code and data are released at github.com/nightingal3/idiom-translation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.933

@inproceedings{liu2023crossing,
  title = {{Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting}},
  author = {Liu, Emmy and Chaudhary, Aditi and Neubig, Graham},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.933},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.933}
}

References

47 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruchit Agrawal, Vighnesh Chenthil Kumar, Vignesh-waran Muralidharan, and Dipti Sharma. 2018. No more beating about the bush : A step towards idiom handling for Indian language NLP. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Jap… find
  2. Dimitra Anastasiou. 2010. Idiom treatment experiments in machine translation. Ph.D. thesis. find
  3. Yehoshua Bar-Hillel. 1952. The treatment of “idioms” by a translating machine. In Proceedings of the Conference on Mechanical Translation, Massachusetts Institute of Technology. find
  4. Christos Baziotis, Prashant Mathur, and Eva Hasler. 2022. Automatic evaluation and analysis of idioms in neural machine translation. find our card
  5. Nicoletta Calzolari, Charles J. Fillmore, Ralph Grishman, Nancy Ide, Alessandro Lenci, Catherine MacLeod, and Antonio Zampolli. 2002. Towards best practice for multiword expressions in computational lexicons. In Proceedings of the Third International Conference on Language Resources and Evaluation (… find
  6. Wan Yu Ho, Christine Kng, Shan Wang, and Francis Bond. 2014. Identifying idioms in Chinese translations. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 716–721, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  7. Santiago Castro. 2017. Fast Krippendorff: Fast computation of Krippendorff’s alpha agreement measure. https://github.com/pln-fing-udela r/fast-krippendorff. ↗
  8. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547. find our card
  9. Thomas C. Cooper. 1999. Processing of Idioms by L2 Learners of English. TESOL Quarterly, 33(2):233– 262. find
  10. Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022a. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4154–4175, Dublin, Ireland… find
  11. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022b. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Irelan… find our card
  12. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  13. George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 451–459, Cambridge, MA. Association for Computational Lingu… find
  14. Yvette Graham, Nikita Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266–274. Association for Computational Linguistics. find
  15. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  16. Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2022. Understanding transformer memorization recall through idioms. find our card
  17. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. find our card
  18. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neighbor machine translation. find
  19. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  20. Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30:61 – 70. find
  21. Lori Levin, Teruko Mitamura, Brian MacWhinney, Davida Fromm, Jaime Carbonell, Weston Feely, Robert Frederking, Anatole Gershman, and Carlos Ramirez. 2014. Resources for the detection of conventionalized metaphors in four languages. In Proceedings of the Ninth International Conference on Language Res… find
  22. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Reso… find our card
  23. Dayan Liu. 2012. Translation and culture: Translating idioms between english and chinese from a cultural perspective. Theory and Practice in Language Studies, 2:2357–2362. find
  24. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica, (12):0455–463. find
  25. Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. find
  26. Evgeny Matusov. 2019. The challenges of using neural machine translation for literature. In Proceedings of the Qualities of Literary Machine Translation, pages 10–19, Dublin, Ireland. European Association for Machine Translation. find
  27. Makoto Nagao. 1984. A framework of a mechanical translation between japanese and english by analogy principle. In Proc. of the International NATO Symposium on Artificial and Human Intelligence, page 173–180, USA. Elsevier North-Holland, Inc. find
  28. Abdulfattah Omar and Yasser Gomaa. 2020. The machine translation of literature: Implications for translation pedagogy. International Journal of Emerging Technologies in Learning (iJET), 15:228. find
  29. Sixth Confrence on Machine Translation (WMT21). Shared task: Large-scale multilingual machine translation. find
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. find our card
  31. Maria Pershina, Yifan He, and Ralph Grishman. 2015. Idiom paraphrases: Seventh heaven vs cloud nine. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 76–82, Lisbon, Portugal. Association for Computational Linguistics. find
  32. Thierry Poibeau. 2022. On "human parity" and "super human performance" in machine translation evaluation. Marseille, France. Language Resource and Evaluation Conference. find
  33. Paul H. Portner. 2005. What is meaning?: Fundamentals of formal semantics. Blackwell Publishing, Malden, MA. find
  34. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. find our card
  35. Andrew Radford. 2004. English Syntax: An Introduction. Cambridge University Press, Cambridge, UK. find
  36. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Unbabel’s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics. find our card
  37. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
  38. Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for nlp. In Computational Linguistics and Intelligent Text Processing, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg. find our card
  39. Giancarlo Salton, Robert Ross, and John Kelleher. 2014. Evaluation of a substitution method for idiom transformation in statistical machine translation. In Proceedings of the 10th Workshop on Multiword Expressions (MWE), pages 38–42, Gothenburg, Sweden. Association for Computational Linguistics. find
  40. Yutong Shao, Rico Sennrich, Bonnie Webber, and Federico Fancellu. 2017. Evaluating machine translation performance on chinese idioms with a blacklist method. find
  41. Antonio Toral and Andy Way. 2018. What level of quality can neural machine translation attain on literary text? In J Moorkens, S Castilho, F Gaspari, and S Doherty, editors, Translation Quality Assessment: Technologies and Applications. Springer, Cham. find
  42. Unbabel. 2019. Why translating idioms is hard. find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  44. Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482–1488, Copenhagen, Denmark. Association for Computation… find
  45. Eric Wehrli. 1998. Translating idioms. In COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics. find
  46. Chunli Yang. 2010. Cultural differences on chinese and english idioms of diet and the translation. English Language Teaching, 3. find
  47. Andrea Zaninello and Alexandra Birch. 2020. Multiword expression aware neural machine translation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3816–3825, Marseille, France. European Language Resources Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-crossing-threshold-idiomatic-machine-translation/