Elamur Research⌕
☰
home / papers / 2015 / deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

Paper · 2015 · Michel Galley, Chris Brockett, Alessandro Sordoni et al.

Summary

Automatic metrics like BLEU work reasonably well for machine translation. They struggle when many different outputs are acceptable, as in conversational response generation. There, natural data offer only one reply per message, and judging relevance is subjective.

Galley et al. propose ∆BLEU (Discriminative BLEU), a corpus-level multi-reference BLEU in which each reference carries a human-assigned quality weight in [−1, +1]. An n-gram match earns credit according to the highest-weighted reference that contains it, so overlap with poor references lowers the score. With all weights set to 1, the metric becomes standard BLEU, and its maximum stays at 1.

To build test data, the authors used BM25 retrieval to mine extra candidate replies from a large Twitter corpus. Crowd workers rated these replies, and all of them were kept with rescaled weights. Compared against human judgments of system pairs, ∆BLEU tracked human preferences clearly better than IBM BLEU and sentence-level BLEU. The approach moves human effort into a one-time rating of references, which can then be reused throughout development.

Key points

  • Data source: 29M Twitter context–message–response triples. About 33K were crowd-rated, and the 4232 triples averaging ≥4 were split into seed dev (2118) and test (2114) sets.
  • For each seed pair, the top 15 BM25-retrieved responses were added. Each was rated by 5 annotators on a 5-point scale, and scores were mapped to [−1, +1] without any cutoff.
  • The human evaluation covered outputs from 7 systems (12 system pairs), yielding about 126K pairwise ratings. All metrics used BLEU-2.
  • With all references, ∆BLEU reached Spearman's ρ = .484 and Kendall's τ = .342. The best IBM BLEU scores were .343 and .232 (w ≥ 0.6), and sentence-level BLEU peaked at .330 and .222.
  • IBM BLEU got worse as low-rated references were added, while ∆BLEU improved, even slightly benefiting from negatively weighted references. The code, multi-reference dev/test sets, and rated outputs were released by Microsoft Research.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🏢MetaMenlo Park · United States214 works · 241,492 citations🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao and Bill Dolan. 2015. deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets. arXiv:1506.06863.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.06863 · arXiv 1506.06863

@misc{galley2015deltableu,
  title = {{deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets}},
  author = {Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao and Bill Dolan},
  year = {2015},
  doi = {10.48550/arxiv.1506.06863},
  eprint = {1506.06863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.06863}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. M ETEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. find
  2. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of B LEU in machine translation research. In EACL, pages 249–256. find
  3. Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proc. of MT Summit IX, pages 63– 70. find
  4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. of HLT, pages 138–145. find
  5. Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evaluation. In Proc. of HLT-NAACL, pages 162–171. find
  6. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proc. of EMNLP, pages 172–176.
  7. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proc. of NAACL-HLT, pages 1183–1191.
  8. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853–899. find
  9. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proc. of the 22nd ACM International Conference on Information & Knowledge Management, pages 2333–2338. find
  10. Alon Lavie and Abhaya Agarwal. 2007. M ETEOR : An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proc. of the Workshop on Statistical Machine Translation (StatMT), pages 228–231. find
  11. Preslav Nakov, Francisco Guzman, and Stephan Vogel. 2012. Optimizing for Sentence-Level BLEU+1 Yields Short Translations. In Proc. of COLING. find
  12. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, pages 160–167. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. B LEU : a method for automatic evaluation of machine translation. In Proc. of ACL, pages 311–318. find
  14. M. Przybocki, K. Peterson, and S. Bronsart. 2008. Official results of the NIST 2008 ”Metrics for MAchine TRanslation” challenge (MetricsMATR08). http://nist.gov/speech/tests/metricsmatr/2008/results/. ↗
  15. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proc. of EMNLP, pages 583–593.
  16. Stephen E Robertson, Steve Walker, Susan Jones, et al. 1995. Okapi at TREC-3. In TREC.
  17. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proc. of NAACL-HLT.
  18. Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In ACL, pages 38–42. find
  19. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/galley-2015-deltableu-discriminative-metric-generation-tasks/