MENLI: Robust Evaluation Metrics from Natural Language Inference

Paper · 2023 · Yanran Chen and Steffen Eger

Summary

Popular BERT-based metrics for text generation, such as BERTScore and MoverScore, score candidates by how semantically close they are to a reference or source. Chen and Eger argue that this is the wrong objective. A translation can be very similar to the reference and still contain wrong numbers, names or facts. They propose MENLI, a family of metrics that take pre-trained NLI models off the shelf and score candidates by entailment (bi-directional for references, source-to-summary for summarization). The work covers MT and summarization, in both reference-based and reference-free settings.

To test robustness, the authors build a preference-based adversarial suite that needs no human annotators. Each item asks whether a metric prefers a correct paraphrase over a fluent but factually corrupted candidate.

Main results:

NLI metrics are far more robust than standard metrics.
On standard benchmarks they beat existing summarization metrics but trail SOTA MT metrics.
Combining NLI metrics with existing ones improves both robustness and correlation with human judgments.

This suggests that inference-based modeling is a promising basis for trustworthy evaluation.

Key points

On the adversarial suite, BLEU, ROUGE, MoverScore and BERTScore perform at or below chance. In MT, NLI metrics beat the best unsupervised standard metric by roughly 30–50%.
Ensembles of standard and NLI metrics gain about 15–30% in adversarial robustness and about 5–30% on standard benchmarks. One exception: cross-lingual (reference-free MT) NLI metrics are weak, so standard metrics make better ensemble partners there.
On WMT20 de, RoMe averages 43% accuracy versus over 80% for the NLI metrics. On BAGEL both reach about 0.23 Spearman's ρ.
Sentence-level NLI variants for summarization scored lower than plain NLI-R on adversarial data (NLI-R 72%):
- SummaCZS: 53%
- FalseSum-tuned models: about 55–68%
Code and data are available at github.com/cyr19/MENLI. Choosing the pooling strategy on held-out datasets changes results only slightly (about −1.9% to +0.8%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓Technische Universität DarmstadtDarmstadt ·

Germany109 works · 35,920 citations 🎓Bielefeld UniversityBielefeld ·

Germany35 works · 1,196 citations

Fields, topics and concepts

📚 Shelves 4Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works

🧭 Topics 2Reasoning · 1,426 works Evaluation & benchmarks · 1,220 works

Cite

Cite as Yanran Chen and Steffen Eger. 2023. MENLI: Robust Evaluation Metrics from Natural Language Inference. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00576

@article{chen2023menli,
  title = {{MENLI: Robust Evaluation Metrics from Natural Language Inference}},
  author = {Yanran Chen and Steffen Eger},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00576},
  url = {https://doi.org/10.1162/tacl_a_00576}
}

References

1 references, 1 of them with a card of ours. ↗ leads to the work.

Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 256–262, Minneapolis, Min…

Updated 31 October 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-menli-robust-evaluation-metrics-natural/