Elamur Research⌕
☰
home / papers / 2019 / compare-mt: A Tool for Holistic Comparison of Language Generation Systems

compare-mt: A Tool for Holistic Comparison of Language Generation Systems

Summary

Corpus-level metrics such as BLEU show whether one generation system beats another, but not what actually changed in the output. Manual inspection of examples is slow and prone to confirmation bias, and one-off analysis scripts take effort and require knowing in advance what to look for. The authors present compare-mt, an open-source pure-Python tool that takes a reference file and two or more system outputs and produces an aggregate report on how the systems differ. The report covers:

  • overall scores with bootstrap confidence intervals and significance tests;
  • word accuracy grouped into buckets (e.g., by training frequency);
  • sentence statistics grouped by length, length difference or score;
  • n-grams that are characteristic of each system;
  • the sentences where the systems diverge most.

Optional inputs add word labels (such as POS tags), source text with alignments, and per-word log likelihoods. On a Slovak-English PBMT vs. NMT comparison, the tool surfaces concrete contrasts, such as NMT's weakness on rare words. These findings then guide closer inspection of individual examples. The tool has already supplied analyses for several published papers.

Key points

  • Basic usage is a single command, compare-mt ref sys1 sys2. Results go to the terminal or to an HTML report with charts and LaTeX tables. Code is at github.com/neulab/compare-mt.
  • Supported metrics include BLEU, chrF, RIBES and output/reference length ratio. Confidence intervals and pairwise significance come from bootstrap resampling (Koehn, 2004).
  • In the running example (Neubig and Hu, 2018 systems):
    • NMT scored higher BLEU (24.03 vs. 22.43 for PBMT, p<0.001).
    • NMT produced shorter outputs (length ratio 93.82 vs. 94.79).
    • RIBES showed no significant difference (both 80.00, p=0.44).
  • Bucketed findings from the same comparison:
    • PBMT handled low-frequency words more robustly and did better on very long sentences.
    • NMT was slightly stronger on very frequent words and very short sentences, but sometimes generated extremely short outputs.
    • POS bucketing showed PBMT was better at base-form verbs and NMT at conjugated verbs.
  • The tool is extensible through user-defined Scorer and Bucketer classes. A companion tool, compare-ll, applies the same bucketed analysis to per-word log likelihoods.

Interesting ideas

  • I1. Characteristic n-grams via a smoothed ratio. Each n-gram is scored by the share of its reference matches that come from system 1 versus system 2, with a smoothing constant α. A large α favors frequent n-grams with reliable counts; a small α favors n-grams that are highly specific to one system. This works for contrasting any two text sources, such as two corpora or two taggers' outputs. In the paper, this analysis exposed a PBMT misspelling ("hemispher") inherited from a bad alignment.
  • I2. Bucket type and statistic as independent settings. Choosing what to group by (length, length difference, score) separately from what to compute (score, count) yields many analyses from one simple design. The pattern suits evaluation dashboards in any NLP project.
  • I3. Labels supplied as external parallel files. Word labels come from files with one label per token, so POS tags, morphological features or lexicon membership can be added without touching the tool's code. This would be convenient for morphologically rich languages like Ukrainian, where accuracy could be broken down by UD morphological features.
  • I4. Report precision and recall separately, not only F-measure. Splitting the two shows whether a system over-produces or under-produces a given class of words. Plain accuracy hides this distinction.
  • I5. Rank sentences by score difference. Sorting test sentences by the per-sentence metric gap between systems gives a principled list of examples to inspect manually. It links aggregate statistics to concrete cases and reduces cherry-picking driven by prior assumptions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging · Machine translation

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-4007

@inproceedings{neubig2019compare,
  title = {{compare-mt: A Tool for Holistic Comparison of Language Generation Systems}},
  author = {Neubig, Graham and Dou, Zi-Yi and Hu, Junjie and Michel, Paul and Pruthi, Danish and Wang, Xinyi},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)},
  year = {2019},
  doi = {10.18653/v1/n19-4007},
  url = {https://doi.org/10.18653/v1/n19-4007}
}

References

52 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. Terrorcat: a translation error categorization-based mt quality metric. In Proc. WMT, pages 64–70. find
  2. Koichi Akabe, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Discriminative language models as a tool for machine translation error analysis. In Proc. COLING, pages 1124–1132. find
  3. Mary Flanagan. 1994. Error classification for mt evaluation. In Proc. AMTA, pages 65–72. find
  4. Wilker Aziz, Sheila Castilho, and Lucia Specia. 2012. Pet: a tool for post-editing and assessing machine translation. In Proc. LREC, pages 3982–3987. find our card
  5. Meritxell Gonzàlez, Jesús Giménez, and Lluı́s Màrquez. 2012. A graphical interface for mt evaluation and error analysis. In Proceedings of the ACL 2012 System Demonstrations, pages 139–144. find
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  7. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proc. EMNLP, pages 2476–2486. find our card
  8. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proc. NAACL, pages 1304–1313, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  9. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proc. EMNLP, pages 944–952. find our card
  10. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proc. EMNLP, pages 257–267, Austin, Texas. Association for Computational Linguistics. find our card
  11. Ondřej Klejch, Eleftherios Avramidis, Aljoscha Burchardt, and Martin Popel. 2015. Mt-compareval: Graphical evaluation interface for machine translation development. The Prague Bulletin of Mathematical Linguistics, 104(1):63–74. find
  12. Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for mt evaluation: evaluating reordering. Machine Translation, 24(1):15–26. find
  13. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. EMNLP, pages 388–395. find
  14. Konstantinos Chatzitheodorou and Stamatis Chatzis-tamatis. 2013. COSTA MT evaluation tool: An open toolkit for human machine translation evaluation. The Prague Bulletin of Mathematical Linguistics, 100(1):83–89. find
  15. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge Press. find
  16. Phillip Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. HLT, pages 48–54. find
  17. David Chiang, Adam Lopez, Nitin Madnani, Christof Monz, Philip Resnik, and Michael Subotin. 2005. The hiero machine translation system: Extensions, evaluation, and analysis. In Proc. EMNLP, pages 779–786. find
  18. Sachin Kumar and Yulia Tsvetkov. 2019. Von mises-fisher loss for training sequence to sequence models with continuous outputs. In Proc. of ICLR. find
  19. Steve DeNeefe, Kevin Knight, and Hayward H. Chan. 2005. Interactively exploring a machine translation model. In Proc. ACL, pages 97–100. Association for Computational Linguistics. find
  20. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121–126, Copenhagen, Denmark. Associa… find
  21. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proc. WMT, pages 85–91. find our card
  22. Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1150–1159. find
  23. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81. find
  24. Ahmed El Kholy and Nizar Habash. 2011. Automatic error analysis for morphologically rich languages. In Proc. MT Summit, pages 225–232. find
  25. Adam Lopez and Philip Resnik. 2005. Pattern visualization for machine translation output. In Proceedings of HLT/EMNLP 2005 Interactive Demonstrations, pages 12–13. find
  26. Christian Federmann. 2012. Appraise: an open-source toolkit for manual evaluation of mt output. The Prague Bulletin of Mathematical Linguistics, 98(1):25–35. find
  27. Nitin Madnani. 2011. ibleu: Interactively debugging and scoring statistical machine translation systems. In 2011 IEEE Fifth International Conference on Semantic Computing, pages 213–214. IEEE. find
  28. Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press. find
  29. Maja Popović and Hermann Ney. 2007. Word error rates: Decomposition over POS classes and applications for error analysis. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 48–55. find
  30. Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium. find our card
  31. Maja Popović and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657–688. find
  32. Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daume III. 2012. Midge: Generating image descriptions from computer vision detections. In Proc. EACL, pages 747–756. find
  33. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proc. NAACL, New Orleans, USA. find our card
  34. Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. Journal of Artificial Intelligence Research, 55:95–130. find
  35. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press. find
  36. Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional translation models. In Proc. WMT, Brussels, Belgium. find
  37. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proc. WMT, pages 61–72, Belgium, Brussels. Association for Computational Linguistics. find our card
  38. Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs. In Proc. EACL, pages 376–382, Valencia, Spain. find our card
  39. Masaki Murata, Kiyotaka Uchimoto, Qing Ma, Toshiyuki Kanamaru, and Hitoshi Isahara. 2005. Analysis of machine translation systems’ errors in tense, aspect, and modality. In Proc. PACLIC. find
  40. Sara Stymne. 2011. BLAST: A tool for error analysis of machine translation output. In Proceedings of the ACL-HLT 2011 System Demonstrations, pages 56– 61. find our card
  41. Sudip Kumar Naskar, Antonio Toral, Federico Gaspari, and Andy Way. 2011. A framework for diagnostic evaluation of mt based on linguistic checkpoints. Proc. MT Summit, pages 529–536. find
  42. David Vilar, Jia Xu, Luis Fernando d’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. LREC, pages 697– 702. find our card
  43. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Proc. EMNLP, Brussels, Belgium. find our card
  44. Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. 2018. A tree-based decoder for neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium. find
  45. Alice H Oh and Alexander I Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In Proceedings of the 2000 ANLP/NAACL Workshop on Conversational systems-Volume 3, pages 27–32. Association for Computational Linguistics. find
  46. Jonathan Weese and Chris Callison-Burch. 2010. Visualizing data structures in parsing-based machine translation. The Prague Bulletin of Mathematical Linguistics, 93:127–136. find
  47. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318. find our card
  48. Daniel Zeman, Mark Fishel, Jan Berka, and Ondřej Bojar. 2011. Addicter: What is wrong with my translations? The Prague Bulletin of Mathematical Linguistics, 96(1):79–88. find
  49. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  50. Ming Zhou, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. In Proc. COLING, pages 1121–1128, Manchester, UK. find
  51. Coling 2008 Organizing Committee. find
  52. Maja Popović, Adrià de Gispert, Deepa Gupta, Patrik Lambert, Hermann Ney, José B. Mariño, Marcello Federico, and Rafael Banchs. 2006. Morphosyntactic information for automatic error analysis of statistical machine translation output. In Proc. WMT, pages 1–6. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neubig-2019-compare-mt-tool-holistic-comparison/