Elamur Research
☰
home / papers / 2018 / RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6456.pdf

Summary

Most automatic machine translation evaluation (MTE) metrics, such as SentBLEU, chrF++, MEANT 2.0 and Blend, compare a hypothesis with its reference through character or word N-gram overlap. That approach misses sentence-wide meaning. The authors present RUSE, a supervised segment-level metric for to-English translation submitted to the WMT18 metrics shared task. RUSE avoids training sentence encoders on scarce human-judged MT data. It instead reuses pre-trained universal sentence embeddings from InferSent, Quick-Thought and Universal Sentence Encoder. The hypothesis and reference vectors are combined into one feature vector, and a regressor (an MLP or an SVR) learns to predict direct assessment (DA) scores. Training uses DA data from WMT15–WMT17. On WMT16 and WMT17, RUSE outperforms the strongest earlier metrics, including the ensemble metrics DPMF comb and Blend, at both segment and system level. The results show that general-purpose embeddings learned from large corpora transfer well to MT quality estimation without hand-engineered features.

Key points

  • Features: For each embedding type, RUSE concatenates the hypothesis and reference vectors, their element-wise product and their absolute difference, giving a 4d-dimensional input. The embedding sizes are InferSent 4096, Quick-Thought 4800 and USE 512.
  • Training data: The train/dev splits use a 9:1 ratio:
    • 1800/200 from WMT15 for testing on WMT16;
    • 4824/536 from WMT15–16 for testing on WMT17;
    • 8352/928 from WMT15–17 for the WMT18 submission.
  • Segment-level results: Average Pearson correlation on WMT16 is 0.685 (MLP) against 0.633 for DPMF comb. On WMT17 it is 0.682 (MLP) and 0.685 (SVR) against 0.633 for Blend.
  • System-level results: On WMT17, RUSE (MLP) averages 0.975 against 0.960 for Blend. In the ablation, Quick-Thought alone reaches 0.974, which suggests the method could extend to target languages other than English.
  • Submitted model: The MLP version (built with Chainer) usually beat the SVR and runs faster on a GPU, so it was submitted with all three embedding types. The code is published at github.com/Shi-ma/RUSE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

51 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 45 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6456

@inproceedings{shimanaka2018ruse,
  title = {{RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation}},
  author = {Shimanaka, Hiroki and Kajiwara, Tomoyuki and Komachi, Mamoru},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6456},
  url = {https://doi.org/10.18653/v1/w18-6456}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chi-Kiu Lo. 2017. MEANT 2.0: Accurate Semantic MT Evaluation for Any Output Language. In Proceedings of the Second Conference on Machine Translation, pages 589–597. find our card
  2. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 Metrics Shared Task. In Proceedings of the Second Conference on Machine Translation, pages 489–513. find our card
  3. Lajanugen Logeswaran and Honglak Lee. 2018. An Efficient Framework for Learning Sentence Representations. In International Conference on Learning Representations. find our card
  4. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation, pages 199– 231. find our card
  5. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a Novel Combined MT Metric Based on Direct Assessment ― CASICT-DCU submission to WMT17 Metrics Task. In Proceedings of the Second Conference on Machine Translation, pages 598–603. find
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. find our card
  8. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder. arXiv preprint arXiv:1803.11175v2. ↗ our card
  9. Maja Popović. 2017. chrF++: Words Helping Character N-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. find our card
  10. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680. find our card
  11. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 106–111. find
  12. Jesús Giménez and Lluı́s Màrquez. 2010. Asiya: An Open Toolkit for Automatic Machine Translation (Meta-) Evaluation. The Prague Bulletin of Mathematical Linguistics, (94):77–86. find
  13. Miloš Stanojević, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273. find
  14. Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015a. ReVal: A Simple and Effective Machine Translation Evaluation Metric Based on Recurrent Neural Networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1066–1072. find
  15. Miloš Stanojević and Khalil Sima’an. 2015. BEER 1.1: ILLC UvA Submission to Metrics and Tuning Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 396–401. find
  16. Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015b. Machine Translation Evaluation using Recurrent Neural Networks. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 380–384. find
  17. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1556–1566. find
  18. Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013. UMBC EBIQUITY-CORE: Semantic Textual Similarity Systems. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 44–52. Association for Computational Linguistics. find
  19. Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTER: Translation Edit Rate on Character Level. In Proceedings of the First Conference on Machine Translation, pages 505–510. find our card
  20. Chin-Yew Lin and Franz Josef Och. 2004. OR-ANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507. find
  21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
  22. Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015a. CASICT-DCU Participation in WMT2015 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 417–421. find
  23. Hui Yu, Xiaofeng Wu, Wenbin Jiang, Qun Liu, and Shouxun Lin. 2015b. An Automatic Machine Translation Evaluation Metric Based on Dependency Parsing Model. arXiv preprint arXiv:1508.01996. ↗
  24. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27. find our card