Elamur Research
☰
home / papers / 2019 / Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1483.pdf

Summary

Table-to-text systems are usually scored with BLEU or ROUGE. Both metrics assume the reference description is the ideal output. In heuristically collected corpora such as WikiBio, however, references often contain facts absent from the table and leave out facts that are present. Purely reference-based metrics therefore punish correct outputs. The authors introduce PARENT, a metric named for computing precision and recall over n-grams that the table entails, which scores a generation against both the reference and the source table. A lightweight lexical entailment model, based on word overlap or co-occurrence, estimates whether an n-gram is supported by the table. Precision credits n-grams that appear in the reference or are entailed by the table. Recall down-weights unsupported reference n-grams and adds coverage of table records via longest common subsequence. In a large human study of WikiBio models, PARENT tracked human ratings better than existing text generation metrics. It matched information-extraction metrics modeled on Wiseman et al. (2017) while being simpler to apply. It also worked on WebNLG, whose references are written by humans. This makes evaluation more trustworthy for noisy, automatically built datasets.

Key points

  • Human evaluation covered outputs of 16 table-to-text models on 1100 WikiBio examples. Metrics were compared by Pearson correlation with the human scores at the system level, and Spearman correlation showed similar trends.
  • Divergence is widespread: about 62% of WikiBio references mention information that cannot be found in the table.
  • PARENT combines entailed precision and recall into an F-score. Precision and the reference-side recall use n-gram orders 1–4. Recall is a geometric mix of reference recall and table recall, weighted by λ. λ can be tuned on human judgments or set per instance from the reference's own table coverage, which gives an average of about 0.6 on WikiBio. Smoothing uses ε = 10⁻⁵.
  • For the IE-based baselines, a pointer-generator network was trained to convert WikiBio text into a linearized table, reaching 35.1% F-score. It was used to compute the CS, RG and RG-F metrics.
  • Code and data are released by the authors. The metric is also shown to apply to the WebNLG challenge data, where references are of high quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1483

@inproceedings{dhingra2019handling,
  title = {{Handling Divergent Reference Texts when Evaluating Table-to-Text Generation}},
  author = {Dhingra, Bhuwan and Faruqui, Manaal and Parikh, Ankur and Chang, Ming-Wei and Das, Dipanjan and Cohen, William},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1483},
  url = {https://doi.org/10.18653/v1/p19-1483}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Srinivas Bangalore, Owen Rambow, and Steve Whittaker. 2000. Evaluation metrics for generation. In Proc. of INLG. find
  2. Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proc. of EMNLP. find
  3. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of nlg systems. In Proc. of EACL. find
  4. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proc. of WMT. find
  5. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in machine translation research. In Proc. of EACL. find
  6. Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proc.of Workshop on NMT. find
  7. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proc. of EMNLP. find our card
  8. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proc. of ACL. find
  9. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. find
  10. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proc. of WMT. find our card
  11. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proc. of ACL. find
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelber… find
  13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of Workshop on Text Summarization Branches Out. find
  14. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proc. of WMT. find our card
  15. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. of HLT. find
  16. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG Challenge. arXiv preprint arXiv:1901.11528. — [arXiv:1901.11528](https://arxiv.org/abs/1901.11528) find
  17. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for micro-planners. In Proc. of ACL. find
  18. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65–170. find
  19. Oren Glickman and Ido Dagan. 2005. A probabilistic setting and lexical cooccurrence model for textual entailment. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 43–48. Association for Computational Linguistics. find
  20. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proc. of EMNLP. find
  21. Lifu Huang, Avirup Sil, Heng Ji, and Radu Florian. 2017. Improving slot filling performance with attentive neural networks on dependency structures. find
  22. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proc. of EACL. find
  23. Karen Kukich. 1983. Design of a knowledge-based report generator. In Proc. of ACL. find
  24. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2010. PEM: A paraphrase evaluation metric exploiting parallel texts. In Proc. of EMNLP. find
  25. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proc. of EMNLP. find
  26. Kathleen R. McKeown. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, New York, NY, USA. find
  27. Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proc. of EMNLP. find
  28. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017a. Why we need new evaluation metrics for NLG. In Proc. of EMNLP. find
  29. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b. The E2E dataset: New challenges for end-to-end generation. In Proc. of SIGDIAL. find our card
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proc. of ACL. find our card
  31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  32. Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, pages 1–12. find our card
  33. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Nat. Lang. Eng., 3(1):57–87. find
  34. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. EMNLP. find our card