Elamur Research⌕
☰
home / papers / 2020 / Evaluation of Text Generation: A Survey

Evaluation of Text Generation: A Survey

Paper · 2020 · Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao

Summary

The survey addresses the question of how to evaluate natural language generation (NLG) systems. This is difficult because most such tasks are open-ended: many acceptable outputs can be given for a single input. The authors do not conduct experiments of their own. Instead, they systematize the evaluation methods that have emerged in recent years, primarily for neural models based on seq2seq and Transformer. The methods are divided into three groups: human evaluation; automatic metrics that require no training (n-gram overlap, string distances, lexical diversity); and metrics that are themselves trained models and imitate a human evaluator. For each group, achievements and open problems are described. The role of shared tasks is considered separately, along with two examples of task-specific evaluation: automatic summarization and long text generation. The main conclusion is as follows: human evaluation remains the gold standard, but it is expensive, and automatic metrics do not always agree with human judgments. The survey is useful as a map of metrics for those who build and compare generative models, including Ukrainian-language NLP projects.

Key points

  • Metrics are divided into three categories: human evaluation (ratings, pairwise comparisons, the Turing test), untrained automatic metrics, and machine-learned metrics.
  • The survey covers summarization, machine translation, dialogue systems, paraphrasing, question generation, data-to-text, story and poetry generation, and image and video captioning.
  • Two cases are examined in detail: the evaluation of automatic summarization and of long text generation. A separate section is devoted to shared tasks.
  • Future directions: explainable evaluation; higher-quality corpora with multiple reference texts; standardization (NLTK, spaCy, Evaluating Natural Language Generation Challenge); transparent reporting of human evaluation; assessment of ethical risks (example: XiaoIce).
  • The risks of large models such as GPT-2/3 are mentioned separately: hallucinations, non-factual content, and the reproduction of biases from the training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓University of WashingtonSeattle · United States402 works · 212,763 citations🏢MetaMenlo Park · United States214 works · 241,492 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksText generation · 214 works

Cite

Cite as Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao. 2020. Evaluation of Text Generation: A Survey. arXiv:2006.14799.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.14799 · arXiv 2006.14799

@misc{celikyilmaz2020evaluation,
  title = {{Evaluation of Text Generation: A Survey}},
  author = {Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao},
  year = {2020},
  doi = {10.48550/arxiv.2006.14799},
  eprint = {2006.14799},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.14799}
}

References

320 references, 111 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017). find
  2. Vancouver, Canada, August 2017. Association for Computational Linguistics. find
  3. Abhaya Agarwal and Alon Lavie. Meteor, mbleu and mter: Evaluation metrics for high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT ’08, pp. 115–118, USA, 2008. Association for Computational Linguistics. IS… find
  4. Eneko Agirre, Aitor Gonzalez-Agirre, Inigo Lopez-Gazpio, Montse Maritxalar, German Rigau, and Larraitz Uria. Semeval-2016 task 2: Interpretable semantic textual similarity. pp. 512–524, 01 2016. find
  5. Manex Agirrezabal, Bertol Arrieta, Aitzol Astigarraga, and Mans Hulden. POS-tag based poetry generation with WordNet. In Proceedings of the 14th European Workshop on Natural Language Generation, pp. 162–166, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https://www.acl… ↗
  6. Jorge Agnese, Jonathan Herrera, Haicheng Tao, and Xingquan Zhu. A survey and taxonomy of adversarial neural networks for text-to-image synthesis. In WIREs Data Mining and Knowledge Discovery. Wiley, Feb 2020. doi: 10.1002/widm.1345. URL http://dx.doi.org/10.1002/widm.1345. ↗
  7. Ramiz Aliguliyev. Using the f-measure as similarity measure for automatic text summarization. In Vychislitel’nye Tekhnologii, volume 13, 01 2008. find
  8. Jacopo Amidei, Paul Piwek, and Alistair Willis. Rethinking the agreement in human evaluation tasks. In COLING, 2018. find
  9. Jacopo Amidei, Paul Piwek, and Alistair Willis. Agreement is overrated: A plea for correlation to assess human evaluation reliability. In INLG, 2019a. find
  10. Jacopo Amidei, Paul Piwek, and Alistair Willis. The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations. In INLG, 2019b. URL https://www.inlg2019.com/assets/papers/57_Paper.pdf. ↗
  11. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In ECCV, 2016. URL http://arxiv.org/abs/1607.08822. ↗
  12. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. January 2019. 5th International Conference on Learning Representations, ICLR 2017 ; Conference date: 24-04-2017 Through 26-04-2017. find
  13. Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. In Computational Linguistics, volume 34, pp. 555–596, 2008. find
  14. N. Asher and A. Lascarides. Logics of Conversation. Cambridge University Press, 2003. find
  15. W. Aziz, Sheila Castilho, and Lucia Specia. Pet: a tool for post-editing and assessing machine translation. In LREC, 2012.
  16. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
  17. Shuang Bai and Shan An. Va survey on automatic image caption generation. In Neuro Computing, pp. 291–304, 10 2018. find
  18. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, p…
  19. Mousumi Banerjee, Michelle Hopkins Capozzoli, Laura A. McSweeney, and Debajyoti Sinha. Beyond kappa: A review of interrater agreement measures. In The Canadian Journal of Statistics, 1999. find
  20. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, Ann Arbor, Michigan, June 2005.…
  21. Eva Banik, Claire Gardent, and Eric Kow. The KBGen challenge. In Proceedings of the 14th European Workshop on Natural Language Generation, pp. 94–97, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W13-2111. ↗
  22. Adrien Barbaresi. Ad hoc and general-purpose corpus construction from web sources. June 2015. URL https://tel.archives-ouvertes.fr/tel-01167309. ↗
  23. Ellen Gurman Bard, Dan Robertson, and Antonella Sorace. Magnitude estimation of linguistic acceptability. In Language, volume 72, pp. 32–68. Linguistic Society of America, 1996. URL http://www.jstor.org/stable/416793. ↗
  24. Regina Barzilay and Lillian Lee. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In HLT-NAACL 2003: Main Proceedings, pp. 16–23, 2003.
  25. Anja Belz and Eric Kow. The GREC challenges 2010: Overview and evaluation results. In Proceedings of the 6th International Natural Language Generation Conference, 2010. find
  26. Anja Belz and Ehud Reiter. Comparing automatic and human evaluation of nlg systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/E06-1040. ↗
  27. Anja Belz, Eric Kow, and Jette Viethen. The GREC named entity generation challenge 2009: Overview and evaluation results. In Proceedings of the 2009 Workshop on Language Generation and Summarisation (UCNLG+Sum 2009), pp. 88–98, Suntec, Singapore, August 2009. Association for Computational Linguistic… find
  28. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Associati… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  29. Raffaella Bernardi, Ruket Çakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. In CoRR, volume abs/1601.03896, 2016. URL http://ar… find
  30. Alan Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, Jason Williams, Kai Yu, Steve Young, and Maxine Eskenazi. Spoken dialog challenge 2010: Comparison of live and control test results. pp. 2–7, 01… find
  31. Bernd Bohnet and Robert Dale. Viewing referring expression generation as search. pp. 1004–1009, 01 2005. find
  32. Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT 2018),… find
  33. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  34. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  35. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In CoRR, volume abs/1811.02549, 2018. URL http://arxiv.org/abs/1811.02549. — [arXiv:1811.02549](https://arxiv.org/abs/1811.02549) ↗
  36. Chris Callison-Burch and Miles Osborne. Re-evaluating the role of bleu in machine translation research. In In EACL, pp. 249–256, 2006.
  37. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. Meta- evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pp. 136–158, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL h…
  38. Erion Cano and Ondrej Bojar. Keyphrase generation: A multi-aspect survey. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019. find
  39. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT 2018), July 2018.
  40. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2017), 2017a.
  41. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. In CoRR, volume abs/1708.00055, 2017b. URL http://arxiv.org/abs/1708.00055. ↗
  42. David L. Chen and Raymond J. Mooney. Learning to sportscast: A test of grounded language acquisition. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pp. 128–135, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605582054. doi: 10.1145/1390156… — [doi:10.1145/1390156.1390173](https://doi.org/10.1145/1390156.1390173) find
  43. Guanyi Chen and Kees van Deemter. Lessons from computational modelling of reference production in Mandarin and English. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 263–272, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL http… find
  44. Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. Adversarial text generation via feature mover’s distance. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in N… find
  45. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1657–1668, Vancouver, Canada, July 2017. Association for C… find
  46. Tianlang Chen, Zhongping Zhang, Quanzeng You, Chen Fang, Zhaowen Wang, Hailin Jin, and Jiebo Luo. ”factual” or ”emotional”: Stylized image captioning with adaptive learning and attention, 2018b. find
  47. Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 675–686, Melbourne, Australia, July 2018. Association for Computational L…
  48. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pr…
  49. Julian Chow, Lucia Specia, and Pranava Madhyastha. WMDO: Fluency-based word mover’s distance for machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 494– 500, Florence, Italy, August 2019. Association for Computati… find
  50. Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In ACL, 2019. — Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts
  51. J. Clarke and M. Lapata. Global inference for sentence compression: An integer linear programming approach. In Journal of Artificial Intelligence Research, volume 31, pp. 399–429, 2008. find
  52. Arman Cohan and Nazli Goharian. Revisiting summarization evaluation for scientific articles. In CoRR, volume abs/1604.00400, 2016. URL http://arxiv.org/abs/1604.00400. ↗
  53. Jacob Cohen. A coefficient of agreement for nominal scales. In Educational and Psychological Measurement, volume 20, pp. 37–, 04 1960. find
  54. Jacob Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. In Psychological Bulletin, volume 70, pp. 213–220, 1968. doi: 10.1037/h0026256. URL https://doi.org/10.1037/h0026256. — [doi:10.1037/h0026256](https://doi.org/10.1037/h0026256) ↗
  55. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670–680, Copenhagen, Denmark…
  56. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. Learning to evaluate image captioning. In CoRR, volume abs/1806.06422, 2018. URL http://arxiv.org/abs/1806.06422. — [arXiv:1806.06422](https://arxiv.org/abs/1806.06422) ↗
  57. Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. A comprehensive survey of multilingual neural machine translation, 2020. find
  58. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. Tesla at wmt 2011: Translation evaluation and tunable metric. In Proceedings of WMT, 2011. find
  59. Andrew M. Dai, Christopher Olah, and Quoc V. Le. Document embedding with paragraph vectors. In NeurIPS Deep Learning Workshop, 2015. find
  60. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ICLR, 2020.
  61. Michael J. Denkowski, Chris Dyer, and A. Lavie. Learning from post-editing: Online model adaptation for statistical machine translation. In EACL, 2014. find
  62. Etienne Denoual and Yves Lepage. BLEU in characters: Towards automatic MT evaluation in languages without word delimiters. In Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts, 2005. URL https://www.aclweb.org/anthology/I05-2014. ↗
  63. Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. Evaluation metrics for text summarization. In Computing and Informatics, volume 28/2, 2009. URL http://www.cai.sk/ojs/index.php/cai/article/viewFile/37/24. ↗
  64. Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. Survey on evaluation methods for dialogue systems. In CoRR, volume abs/1905.04071, 2019. URL http://arxiv.org/abs/1905.04071. ↗
  65. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In CoRR, volume abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805. ↗
  66. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. Handling divergent reference texts in table-to-text generation. In Proc. of ACL, 2019. find
  67. Djellel Eddine Difallah, Elena Filatova, and Panagiotis G. Ipeirotis. Demographics and dynamics of mechanical turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, 2018. find
  68. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. pp. 138–145, 01 2002. find
  69. Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), January 2005.
  70. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In CoRR, volume abs/1905.03197, 2019. URL http://arxiv.org/abs/1905.03197. ↗
  71. Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In CoRR, volume abs/1705.00106, 2017. URL http://arxiv.org/abs/1705.00106. ↗
  72. Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5055–5070, Online, July 2020. Association for Computational Lin…
  73. Ondřej Dušek and Zdeněk Kasner. Evaluating semantic accuracy of data-to-text generation with natural language inference. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 131–137, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL… find
  74. Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. Referenceless quality estimation for natural language generation. In CoRR, volume abs/1708.01759, 2017. URL http://arxiv.org/abs/1708.01759. — [arXiv:1708.01759](https://arxiv.org/abs/1708.01759) ↗
  75. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. Findings of the E2E NLG challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pp. 322–328, Tilburg University, The Netherlands, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/… find
  76. Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. In CoRR, volume abs/1901.07931, 2019. URL http://arxiv.org/abs/1901.07931. ↗
  77. Thomas Eckart, Uwe Quasthoff, and Dirk Goldhahn. The influence of corpus quality on statistical measurements on language resources. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pp. 2318–2321, Istanbul, Turkey, May 2012. European Language Resou… find
  78. Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. Automated rationale generation: A technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Mar 2019. ISBN 9781450362726.… — [doi:10.1145/3301275.3302316](https://doi.org/10.1145/3301275.3302316) find
  79. M. Elsner and E. Charniak. Coreference-inspired coherence modeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, 2008.
  80. Matan Eyal, Tal Baumel, and Michael Elhadad. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North. Association for Computational Linguistics, 2019a. URL http://dx.doi.org/10.18653/v1/n19-1395. ↗
  81. Matan Eyal, Tal Baumel, and Michael Elhadad. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor…
  82. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,…
  83. Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In CoRR, volume abs/1805.04833, 2018. URL http://arxiv.org/abs/1805.04833. ↗
  84. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: long form question answering. In CoRR, volume abs/1907.09190, 2019a. URL http://arxiv.org/abs/1907.09190. ↗
  85. Angela Fan, Mike Lewis, and Yann N. Dauphin. Strategies for structuring story generation. In CoRR, volume abs/1902.01109, 2019b. URL http://arxiv.org/abs/1902.01109. ↗
  86. JL Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378–382, November 1971. ISSN 0033-2909. doi: 10.1037/h0031619. URL https://doi.org/10.1037/h0031619. — [doi:10.1037/h0031619](https://doi.org/10.1037/h0031619) ↗
  87. Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018.
  88. Saadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli Celikyilmaz, and Yejin Choi. In Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow, 2021. URL http://arxiv.org/abs/1907.01272. — [arXiv:1907.01272](https://arxiv.org/abs/1907.01272) ↗
  89. Michael Gamon. Linguistic correlates of style: authorship classification with deep linguistic analysis features. In Association for Computational Linguistics, 2004. find
  90. Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational ai. In Foundations and Trends® in Information Retrieval, volume 13, pp. 127–298. Now Publishers, Inc., 2019. find
  91. Cristina Garbacea, Samuel Carton, Shiyan Yan, and Qiaozhu Mei. Judge the judges: A large-scale evaluation study of neural language models for online review generation. In CoRR, volume abs/1901.00398, 2019. URL http://arxiv.org/abs/1901.00398. ↗
  92. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pp. 124–133, Santiago de Compostela, Spain, September 2017. Association for Compu…
  93. Albert Gatt and Anja Belz. Attribute selection for referring expression generation: New algorithms and evaluation methods. In Proceedings of the Fifth International Natural Language Generation Conference, INLG ’08, pp. 50–58, USA, 2008. Association for Computational Linguistics. find
  94. Albert Gatt and Emiel Krahmer. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. In Journal of Artificial Intelligence Research, volume 61, pp. 65–170, 2018. find
  95. Albert Gatt, Ielka Van Der Sluis, and Kees Van Deemter. Evaluating algorithms for the generation of referring expressions using a balanced corpus. In In Proceedings of the 11th European Workshop on Natural Language Generation, pp. 07, 2007. find
  96. Albert Gatt, Anja Belz, and Eric Kow. The TUNA challenge 2008: Overview and evaluation results. In Proceedings of the Fifth International Natural Language Generation Conference. Association for Computational Linguistics, 2008. find
  97. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 111–116, Florence, Italy, July 2019. Association for Compu…
  98. Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondvrej Duvsek, Chris C. Emezue, Varun Gangal, Cristina Garbacea, T. Hashimoto, Yuf… — The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
  99. M. Ghazvininejad, X. Shi, Y. Choi, and K. Knight. Generating topical poetry. In EMNLP, 2016. find
  100. Dimitra Gkatzia and Saad Mahamood. A snapshot of NLG evaluation practices 2005 - 2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pp. 57–60, Brighton, UK, September 2015. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W15-4708… ↗
  101. Dimitra Gkatzia, Verena Rieser, Phil Bartie, and William Mackaness. From the virtual to the real world: Referring to objects in real-world spatial scenes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1936–1942. Association for Computational Linguisti… find
  102. Yoav Goldberg, Graeme Hirst, Yang Liu, , and Meng Zhang. Neural network methods for natural language processing. In Computational Linguistics, volume 44(1), 2018. find
  103. Zhengxian Gong, Min Zhang, and Guodong Zhou. Document-level machine translation evaluation with gist consistency and text cohesion. In Association for Computational Linguistics, pp. 33–40, 2015. find
  104. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In arXiv 1406.2661, 2014.
  105. Cyril Goutte. Automatic evaluation of machine translation quality. 2006. find
  106. Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3592–3603, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.322. URL…
  107. Yvette Graham. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 128–137, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL https://www.aclweb.org/anth… ↗
  108. Yvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 172–176, Doha, Qatar, October 2014. Association for Computational Linguistics. URL https://…
  109. Alex Graves. Generating sequences with recurrent neural networks. In CoRR, volume abs/1308.0850, 2013. URL http://arxiv.org/abs/1308.0850. ↗
  110. Najeh Hajlaoui and Andrei Popescu-Belis. Assessing the accuracy of discourse connective translations: Validation of an automatic metric. In Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing, 2013. find
  111. Aaron L.-F Han and Derek Wong. Machine translation evaluation: A survey. https://arxiv.org/abs/1605.04515, 05 2016. ↗
  112. Aaron L.-F Han, Derek Wong, Lidia Chao, Liangye He, Yi Lu, Junwen Xing, and Xiaodong Zeng. Mt summit13.language-independent model for machine translation evaluation with reinforced factors. 09 2013a. find
  113. Aaron L.F. Han, Derek Wong, Lidia Chao, Liangye He, and Yi Lu. Unsupervised quality estimation model for english to german translation and its application in extensive supervised evaluation. In The Scientific World Journal, 2013b. find
  114. Lifeng Han. Machine translation evaluation resources and methods: A survey. In IPRC-2018 - Ireland Postgraduate Research Conference, 2018. find
  115. In Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short…
  116. Helen Hastie and Anja Belz. A comparative evaluation methodology for NLG in interactive systems. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp. 4004–4011, Reykjavik, Iceland, May 2014a. European Language Resources Association (ELRA). URL http… find
  117. Helen F. Hastie and Anja Belz. A comparative evaluation methodology for nlg in interactive systems. In LREC, 2014b. find
  118. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In CoRR, volume abs/1506.03340, 2015. URL http://arxiv.org/abs/1506.03340. ↗
  119. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In CoRR, volume abs/1602.03483, 2016. URL http://arxiv.org/abs/1602.03483. ↗
  120. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural Computation, volume 9, pp. 1735–1780, 1997. find
  121. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, volume abs/1904.09751, 2020.
  122. Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. In CoRR, volume abs/1810.04020, 2018. URL http://arxiv.org/abs/1810.04020. ↗
  123. David M. Howcroft, Anya Belz, Miruna Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, S. Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In INLG, 2020. find
  124. Liang Huang, Kai Zhao, and Mingbo Ma. When to finish? optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017. doi: 10.18653/v1/d17-1227. URL http://d… find
  125. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using click-through data. ACM International Conference on Information and Knowledge Management (CIKM), October 2013. URL https://www.microsoft.com/en-us/research/publ… ↗
  126. Xuedong Huang, Fileno Alleva, Hsiao wuen Hon, Mei yuh Hwang, and Ronald Rosenfeld. The sphinx-ii speech recognition system: An overview. 7:137–148, 1992. find
  127. Text Inspector. Measure lexical diversity, 2013. URL https://textinspector.com/help/lexical-diversity/. ↗
  128. Panagiotis G. Ipeirotis, F. Provost, and Jing Wang. Quality management on amazon mechanical turk. In HCOMP ’10, 2010. find
  129. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 944– 952, Cambridge, MA, October 2010. Association for…
  130. Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. Tiger: Text-to-image grounding for image caption evaluation. In EMNLP 2019, November 2019. find
  131. Shafiq Joty, Francisco Guzman, Lluis Marquez, and Preslav Nakov. Discourse structure in machine translation evaluation. In Computational Linguistics, volume 43(4), pp. 683–722, 2017.
  132. Daniel Jurafsky and James H. Martin. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. In Speech and Language Processing. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2009. find
  133. Filip Jurcicek, Simon Keizer, Milica Gasic, François Mairesse, Blaise Thomson, Kai Yu, and Steve Young. Real user evaluation of spoken dialogue systems using amazon mechanical turk. pp. 3061–3064, 01 2011. find
  134. Sushant Kafle and Matt Huenerfauth. Evaluating the usability of automatically generated captions for people who are deaf or hard of hearing. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 165–174, New York, NY, USA, 2017. Association for Computi… find
  135. Hassan Kané, Yusuf Kocyigit, Pelkins Ajanoh, Ali Abdalla, and Mohamed Coulibali. Towards neural language evaluators. In Neurips 2019 Document Intelligence Workshop, 2019. find
  136. David Kauchak and Regina Barzilay. Paraphrasing for automatic evaluation. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, 2006.
  137. Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. Genie: A leaderboard for human-in-the-loop evaluation of text generation. In ArXiv, volume abs/2101.06561, 2021. — [arXiv:2101.06561](https://arxiv.org/abs/2101.06561) find
  138. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. Re-evaluating automatic metrics for image captioning. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics,… find
  139. Yoon Kim, Sam Wiseman, and Alexander M. Rush. A tutorial on deep latent variable models of natural language. In CoRR, volume abs/1812.06834, 2018. URL http://arxiv.org/abs/1812.06834. ↗
  140. Svetlana Kiritchenko and Saif M. Mohammad. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 811–817, S…
  141. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-thought vectors. In CoRR, volume abs/1506.06726, 2015. URL http://arxiv.org/abs/1506.06726. ↗
  142. D. Knight, K.—Marcu. Statistics-based summarization – step one: Sentence compression. In In Proceeding of The 17th National Conference of the American Association for Artificial Intelligence, pp. 703–710, 2000. find
  143. Alexander Koller, Donna Byron, Justine Cassell, Robert Dale, Johanna Moore, Jon Oberlander, and Kristina Striegnitz. The software architecture for the first challenge on generating instructions in virtual environments. In Proceedings of the Demonstrations Session at EACL 2009, pp. 33–36, Athens, Gre… find
  144. Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. Text generation from knowledge graphs with graph transformers. In ArXiv, volume abs/1904.02342, 2019.
  145. I Konstas and M. Lapara. Unsupervised concept-to-text generation with hypergraphs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 752–761, 2012. find
  146. I. Konstas and M. Lapata. A global model for concept-to-text generation. In Journal of Artificial Intelligence Research, volume 48, pp. 305–346, 2013. find
  147. E. Krahmer and M. Theune. Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation. LNCS sublibrary: Artificial intelligence. Springer, 2010. ISBN 9783642155727. URL https://books.google.com/books?id=aifpm9shAw8C. ↗
  148. Klaus Krippendorff. Estimating the reliability, systematic error and random error of interval data. In Educational and Psychological Measurement, volume 30, pp. 61–70, 1970. find
  149. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc…
  150. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. 2019b.
  151. M. J. Kusner, Y. Sun, N. I. Kolkin, and K. Q. Weinberger. From word embeddings to document distances. In ICML, 2015. find
  152. Lori Lamel, Sophie Rosse, Jean-Luc Gauvain, Samir Bennacef, Matine Garnier-Rizet, and Bernard Prouts. The limsi arise system. In Speech Communication, pp. 339–353, 2000. find
  153. Weiyu Lan, Xirong Li, and Jianfeng Dong. Fluency-guided cross-lingual image captioning. In ACL Multimedia, volume abs/1708.04390, 2017. URL http://arxiv.org/abs/1708.04390. — [arXiv:1708.04390](https://arxiv.org/abs/1708.04390) ↗
  154. Mirella Lapata. Probabilistic text structuring: Experiments with sentence ordering. In proceedings of the annual meeting of the Association for Computational Linguistics, The Association of Computational Linguistics, 2003.
  155. Mirella Lapata and Regina Barzilay. Automatic evaluation of text coherence: Models and representations. In In Kaelbling, L.P., Saffiotti, A., eds.: IJCAI, Professional Book Center, 2005. find
  156. Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pp. 228–231, USA, 2007. Association for Computational Linguistics.
  157. Alon Lavie, Kenji Sagae, and Shyamsundar Jayaraman. The significance of recall in automatic metrics for mt evaluation. In AMTA, 2004. find
  158. Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with application to the biography domain. In CoRR, volume abs/1603.07771, 2016a.
  159. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203–1213, Austin, Texas, November 2016b. Association for Computational…
  160. Audrey J. Lee and Mark A. Przybocki. Nist 2005 machine translation evaluation official results. 2005. find
  161. C. Lee, Albert Gatt, Emiel van Miltenburg, and E. Krahmer. Human evaluation of automatically generated text: Current trends and best practice guidelines. In Comput. Speech Lang., volume 67, pp. 101151, 2021. find
  162. Chris Van Der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In INLG, 2019. URL https://www.inlg2019.com/assets/papers/98_Paper.pdf. ↗
  163. Jinchao Li, Qi Zhu, Baolin Peng, Lars Liden, Runze Liang, Ryuichi Takanobu, Shahin Shayandeh, Swadheen Shukla, Zheng Zhang, Minlie Huang, and Jianfeng Gao. Multi-domain task-oriented dialog challenge ii. In The Ninth Dialog System Technology Challenge, 2020. find
  164. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–1…
  165. Nannan Li and Zhenzhong Chen. Learning compact reward for image captioning. In arXiv 2003.10925, 2020. — [arXiv:2003.10925](https://arxiv.org/abs/2003.10925) find
  166. Sheng Li, Zhiqiang Tao, and Yun Fu. Visual to text: Survey of image and video captioning. In IEEE Transactions on Emerging Topics in Computational Intelligence, volume PP, pp. 1–16, 01 2019. doi: 10.1109/TETCI.2019.2892755. — [doi:10.1109/tetci.2019.2892755](https://doi.org/10.1109/tetci.2019.2892755) find
  167. Zhongyang Li, Xiao Ding, and Ting Liu. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1033–1043, Santa Fe, New Mexico, USA, August 2018. Association f… find
  168. Percy Liang, Michael Jordan, and Dan Klein. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 91–99, Suntec, Singapore, Augu…
  169. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W04-1013. ↗
  170. Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 605–612, Barcelona, Spain, July 2004. URL https:…
  171. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Languag…
  172. Ding Liu and Daniel Gildea. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 25–32, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.…
  173. Feifan Liu and Yang Liu. Correlation between rouge and human evaluation of extractive meeting summaries. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, HLT-Short ’08, pp. 201–204, USA, 2008. Association for Com… find
  174. Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. Generating diverse and descriptive image captions using visual paraphrases. 2019a. find
  175. Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradient optimization of spider. pp. 873–881, 10 2017. find
  176. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL 2019, June 2019b. URL https://www.microsoft.com/en-us/research/publication/multi-task-deep-neural-networks-for-natural-language-understanding-2/. ↗
  177. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. In CoRR, volume abs/1907.11692, 2019c. URL http://arxiv.org/abs/1907.11692. ↗
  178. Chi-Kiu Lo. Meant 2.0: Accurate semantic mt evaluation for any output language. In WMT, 2017.
  179. Chi-kiu Lo. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 507– 513, Florence, Italy, August 2019. Association…
  180. Chi-Kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. Fully automatic semantic mt evaluation. In WMT@NAACL-HLT, 2012. find
  181. Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJvJXZb0W. ↗
  182. Dang Hoang Long, Minh-Tien Nguyen, Ngo Xuan Bach, Le-Minh Nguyen, and Tu Minh Phuong. An entailment-based scoring method for content selection in document summarization. In Proceedings of the Ninth International Symposium on Information and Communication Technology, pp. 122–129, New York, NY, USA, 2… find
  183. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press, 2015. find
  184. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. Towards an automatic turing test: Learning to evaluate dialogue responses. In ACL, 2017. URL http://arxiv.org/abs/1708.07149. ↗
  185. Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. Neural text generation: Past, present and beyond. In CoRR, volume abs/1803.07133, 2018. URL http://arxiv.org/abs/1803.07133. — [arXiv:1803.07133](https://arxiv.org/abs/1803.07133) ↗
  186. Kelvin Luu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. Citation text generation. ArXiv, abs/2002.00317, 2020. — [arXiv:2002.00317](https://arxiv.org/abs/2002.00317) find
  187. Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. A set of recommendations for assessing human–machine parity in language translation. In Journal of Artificial Intelligence Research, volume 67, 03 2020. find
  188. Nabin Maharjan, Rajendra Banjade, Dipesh Gautam, Lasang J. Tamang, and Vasile Rus. DT Team at SemEval-2017 task 1: Semantic similarity using alignments, sentence-level embeddings and Gaussian mixture model output. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017… find
  189. F. Mairesse, M. Gasic, F. Jurcicek, S. Keizer, B. Thompson, K. Yu, and S. Young. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 2010 Conference of the Association for Computational Linguistics, 2010. find
  190. W.C. Mann and S.A. Thompson. Rhetorical structure theory: Description and construction of text structures. In In: Kempen G. (eds) Natural Language Generation. NATO ASI Series (Series E: Applied Sciences), volume 135, 1987. find
  191. Daniel Marcu. From discourse structures to text summaries. In Proceedings of ACL’97/EACL’97 Workshop on Intelligent Scalable Text Summarization, pp. 80–88, 1997. find
  192. A. Martin and M. Przybocki. The nist 1999 speaker recognition evaluation - an overview, 2000. find
  193. Lara J. Martin, Prithviraj Ammanabrolu, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. Event representations for automated story generation with deep neural nets. In CoRR, volume abs/1706.01331, 2017. URL http://arxiv.org/abs/1706.01331. ↗
  194. Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. How decoding strategies affect the verifiability of generated text, 2019. find
  195. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2799–2808, Florence, Italy, July 2019. Association for Computa… find
  196. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Association for Computational Linguistics (ACL 2020), 2020.
  197. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. In ArXiv, volume abs/2005.00661, 2020.
  198. P.M. McCarthy and S. Jarvis. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaces to lexical diversity assessment. In Behaviour Research Methods, volume 42(2), pp. 381–392, 2010. URL https://link.springer.com/article/10.3758/BRM.42.2.381. — [doi:10.3758/brm.42.2.381](https://doi.org/10.3758/brm.42.2.381) ↗
  199. Iain Mccowan, Darren Moore, John Dines, Daniel Gatica-Perez, Mike Flynn, Pierre Wellner, and Herve Bourlard. On the use of information retrieval measures for speech recognition evaluation. 01 2004. find
  200. Kris McGuffie and Alex Newhouse. The radicalization risks of gpt-3 and advanced neural language models. 2020. find
  201. Kathleen R. McKeown. Text generation. using discourse strategies and focus constraints to generate natural language text. In Studies in natural language processing, 1985. find
  202. I. Melamed, Ryan Green, and Joseph Turian. Precision and recall of machine translation. 2003.
  203. Thomas Meyer, Andrei Popescu-Belis, Najeh Hajlaoui, and Andrea Gesmundo. Machine translation of labeled discourse connectives. In Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas, 2012.
  204. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In CoRR, volume abs/1310.4546, 2013. URL http://arxiv.org/abs/1310.4546. ↗
  205. George A. Miller. Wordnet: A lexical database for english. In Association for Computing Machinery, volume 38, pp. 39–41, New York, NY, USA, November 1995.
  206. Tanushree Mitra, Clayton J. Hutto, and Eric Gilbert. Comparing person- and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015. find
  207. Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diversity and quality in text generation models. In CoRR, volume abs/1904.03971, 2019a. URL http://arxiv.org/abs/1904.03971. ↗
  208. Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diversity and quality in text generation models. In NeuralGen Workshop at NAACL 2019, volume abs/1904.03971, 2019b. URL http://arxiv.org/abs/1904.03971. ↗
  209. Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, C… — DART: Open-Domain Structured Data Record to Text Generation
  210. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive summarization with reinforcement learning. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT 2018), July 2018.
  211. Preksha Nema and Mitesh M. Khapra. Towards a better metric for evaluating question generation systems. In CoRR, volume abs/1808.10192, 2018. URL http://arxiv.org/abs/1808.10192. ↗
  212. Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. pp. 145–152, 01 2004. find
  213. Austin Lee Nichols and Jon K. Maner. The good-subject effect: investigating participant demand characteristics. In The Journal of general psychology, volume 135 2, pp. 151–65, 2008. find
  214. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation conference, pp. 265–273, Edinburgh, UK, September 5-8 2016. Association for Computational Linguistics. doi: 10.18653/v1/W16-… find
  215. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2241–2252, Copenhagen, Denmark, September 2017. Association for Computational Lingui…
  216. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. RankME: Reliable human ratings for natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 72–7…
  217. Kenji Ono, Kazuo Sumita, and Seiji Miike. Abstract generation based on rhetorical structure extraction. In Proceedings of the International Conference on Computational Linguistics (COLING’94), 1994.
  218. Daniel M Oppenheimer, Tom Meyvis, and Nicolas Davidenko. Instructional manipulation checks: Detecting satisficing to increase statistical power. In Journal of experimental social psychology, volume 45, pp. 867–872. Elsevier, 2009. find
  219. Martin T. Orne. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. 1962. find
  220. Sebastian Padó, Michel Galley, Dan Jurafsky, and Christoper Manning. Textual entailment features for machine translation evaluation. pp. 37–41, 01 2009. find
  221. Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. Recent advances in neural question generation. 2019. URL http://arxiv.org/abs/1905.08949. — [arXiv:1905.08949](https://arxiv.org/abs/1905.08949) ↗
  222. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2002.
  223. Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: A controlled table-to-text generation dataset, 2020.
  224. Jae Sung Park, Marcus Rohrbach, Trevor Darrell, and Anna Rohrbach. Adversarial inference for multi-sentence video description. In CoRR, volume abs/1812.05634, 2018. URL http://arxiv.org/abs/1812.05634. ↗
  225. Carla Parra Escartı́n, Wessel Reijers, Teresa Lynn, Joss Moorkens, Andy Way, and Chao-Hong Liu. Ethical considerations in NLP shared tasks. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pp. 66–73, Valencia, Spain, April 2017. Association for Computational Linguis… find
  226. Ramakanth Pasunuru and Mohit Bansal. Multi-task video captioning with video and entailment generation. In CoRR, volume abs/1704.07489, 2017. URL http://arxiv.org/abs/1704.07489. ↗
  227. Tom Pelsmaeker and Wilker Aziz. Effective estimation of deep generative language models. In CoRR, volume abs/1904.08194, 2019. URL http://arxiv.org/abs/1904.08194. ↗
  228. Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. Few-shot natural language generation for task-oriented dialog. In arXiv 2002.12328, February 2020. URL https://www.microsoft.com/en-us/research/publication/few-shot-natural-language-generation-for-task-or… ↗
  229. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
  230. Maja Popovic, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. Evaluation without references: Ibm1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 99–103, 07 2011. find
  231. Ratish Puduppully, Li Dong, and Mirella Lapata. Data-to-text generation with content selection and planning. In CoRR, volume abs/1809.00582, 2018. — [arXiv:1809.00582](https://arxiv.org/abs/1809.00582) find
  232. Chris Quirk, Chris Brockett, and William Dolan. Monolingual machine translation for paraphrase generation. In EMNLP, 2004.
  233. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. In URL https://s3-us-west-2.amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018. ↗
  234. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  235. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2016.
  236. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. Plotmachines: Outline-conditioned generation with dynamic plot state tracking. In arxiv, 2020. find
  237. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  238. Katharina Reinecke and Krzysztof Z. Gajos. Labinthewild: Conducting large-scale online experiments with uncompensated samples. In CSCW ’15, 2015. find
  239. Ehud Reiter. An architecture for data-to-text systems. In Proceedings of the Eleventh European Workshop on Natural Language Generation, ENLG ’07, pp. 97–104, USA, 2007. Association for Computational Linguistics. find
  240. Ehud Reiter. A structured review of the validity of BLEU. In Computational Linguistics, volume 44, pp. 393–401, September 2018. URL https://www.aclweb.org/anthology/J18-3002. ↗
  241. Ehud Reiter. Ehud reiter’s blog, 2019. URL https://ehudreiter.com/blog-index/. ↗
  242. Ehud Reiter and Anja Belz. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. In Computational Linguistics, volume 35, pp. 529–558, 2009. find
  243. Ehud Reiter and Robert Dale. Building natural language generation systems. USA, 2000a. Cambridge University Press. ISBN 0521620368. find
  244. Ehud Reiter and Robert Dale. Building applied natural language generation systems. In Cambridge University Press, Cambridge, UK, 2000b. find
  245. Ehud Reiter, Roma Robertson, and Liesl Osman. Lessons from a failure: Generating tailored smoking cessation letters. In Artif. Intell., volume 144, pp. 41–58, 2003. find
  246. Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu, and Ian Davy. Choosing words in computer-generated weather forecasts. In Artificial Intelligence, volume 167, pp. 137 – 169, 2005. find
  247. Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should I trust you?”: Explaining the predictions of any classifier. In CoRR, volume abs/1602.04938, 2016. URL http://arxiv.org/abs/1602.04938. ↗
  248. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In AAAI, 2018. find
  249. Brian Richards. Type/token ratios: what do they really tell us? language, volume 14, pp. 201–9, 07 1987. find
  250. Alan Ritter, Colin Cherry, and Bill Dolan. Data-driven response generation in social media. In Empirical Methods in Natural Language Processing (EMNLP), January 2011. URL https://www.microsoft.com/en-us/research/publication/data-driven-response-generation-in-social-media/. ↗
  251. M. Roemmele, A. Gordon, and R. Swanson. Evaluating story generation systems using automated linguistic analyses. In Workshop on Machine Learning for Creativity, at the 23rd SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2017), 2017. find
  252. Antti-veikko I. Rosti, Spyros Matsoukas, and Richard Schwartz. Improved word-level system combination for machine translation. In In Proc. of ACL 2007, pp. 312–319, 2007.
  253. Y. Rubner, C. Tomasi, and L. Guibas. A metric for distributions with applications to image databases. In IEEE, 1998. find
  254. Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the Fourth Workshop on Vision and Language, pp. 70– 80, Lisbon, Portugal, September 2015. As… find
  255. In Journal of child Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint…
  256. William A. Scott. Reliability of content analysis: The case of nominal scale coding. In The Public Opinion Quarterly, volume 19, pp. 321–325. [Oxford University Press, American Association for Public Opinion Research], 1955. URL http://www.jstor.org/stable/2746450. ↗
  257. João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. Chateval: A tool for chatbot evaluation. In NAACL-HLT, 2019. find
  258. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073– 1083, Vancouver, Canada, July 2017. Association for Co… find
  259. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation, 2020.
  260. Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of GANs for language generation. In ICLR, 2019. URL https://openreview.net/forum?id=rJMcdsA5FX. ↗
  261. Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron C. Courville. Multiresolution recurrent neural networks: An application to dialogue response generation. In CoRR, volume abs/1606.00776, 2016a. URL http://arxiv.org/abs/1606.00776. ↗
  262. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In CoRR, volume abs/1605.06069, 2016b. URL http://arxiv.org/abs/1605.06069. ↗
  263. Lei Sha, Lili Mou, Tianyu Liu, Pascal Poupart, Sujian Li, Baobao Chang, and Zhifang Sui. Order-planning neural text generation from structured data. In CoRR, volume abs/1709.00155, 2017. — [arXiv:1709.00155](https://arxiv.org/abs/1709.00155) find
  264. Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. Learning-based composite metrics for improved caption evaluation. In Proceedings of ACL 2018, Student Research Workshop, pp. 14–20, Melbourne, Australia, July 2018. Association for Computational Linguistics. URL https://www.aclw… ↗
  265. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3239–3254, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-e…
  266. Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K. Reddy. Neural abstractive text summarization with sequence-to-sequence models. In CoRR, volume abs/1812.02303, 2018. URL http://arxiv.org/abs/1812.02303. ↗
  267. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 751–758, Belgium, Brussels, October 2018. Association for Computatio…
  268. Anastasia Shimorina. Human vs automatic metrics: on the importance of correlation design. 2021. find
  269. Abhisek Singh and Wei Jin. Ranking summaries for informativeness and coherence without reference summaries. In Proceedings of the Twenty-Ninth International Florida Artificial Intelligent Research Society Conference, 2016. find
  270. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pp. 223–231, 2006.
  271. M. Sporleder, C.—Lapata. Discourse chunking and its application to sentence compression. In Proceedings of HLT/EMNLP, pp. 257–264, 2005. find
  272. Manfred Stede and Carla Umbach. Dimlex: A lexicon of discourse markers for text generation and understanding. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 2, ACL ’98/COLING ’98, pp.…
  273. Josef Steinberger and Karel Jezek. Evaluation measures for text summarization. In Computing and Informatics, volume 28, pp. 251–275, 01 2009. find
  274. K. Steinberger, J.—Jezek. Sentence compression for the lsa-based summarizer. In Proceedings of the 7th International Conference on Information Systems Implementation and Modelling, pp. 141–148, 2006. find
  275. Kristina Striegnitz, Denis Alexandre, Andrew Gargett, Alexander Garoufi, Konstantina Koller, and Mariet Theune. Report on the second challenge on generating instructions in virtual environments (give-2.5). In Proceedings of 13th European Workshop on Natural Language Generation ENLG, 2011. find
  276. Octavia-Maria Sulea. Recognizing textual entailment in Twitter using word embeddings. In 2nd Workshop on Evaluating Vector-Space Representations for NLP, 2017.
  277. Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In CoRR, volume abs/1907.12412, 2019. URL http://arxiv.org/abs/1907.12412. ↗
  278. Manne Suneetha and Sheerin Fatima. Extraction based automatic text summarization system with hmm tagger. In International Journal of Soft Computing and Engineering, volume ISSN: 2231-2307, pp. 2231–2307, 08 2011. find
  279. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In CoRR, volume abs/1409.3215, 2014. URL http://arxiv.org/abs/1409.3215. ↗
  280. Rachel Tatman. Evaluating text output in nlp: Bleu at your own risk, 2019. URL https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213. ↗
  281. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Generating token-level explanations for natural language inference. In CoRR, volume abs/1904.10717, 2019. URL http://arxiv.org/abs/1904.10717. ↗
  282. Jesús Tomás, Josep Àngel Mas, and Francisco Casacuberta. A quantitative method for machine translation evaluation. In Proceedings of the EACL 2003 Workshop on Evaluation Initiatives in Natural Language Processing: are evaluation methods, metrics and resources reusable?, pp. 27–34, Columbus, Ohio,… find
  283. L. Shen Turian, J. P. and I. D. Melamed. Evaluation of machine translation and its evaluation. In In Proceedings of MT Summit IX, New Orleans, U.S.A., 2003.
  284. A. M. Turing. Computing Machinary and Intelligence. In Mind, volume LIX, pp. 433–460, 1950. URL https://doi.org/10.1093/mind/LIX.236.433. ↗
  285. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, 2019. URL https://www.aclweb.org/anthology/W19-8643. ↗
  286. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998–6008, 2017.
  287. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CoRR, volume abs/1411.5726, 2014. URL http://arxiv.org/abs/1411.5726. ↗
  288. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CoRR, volume abs/1411.4555, 2014. URL http://arxiv.org/abs/1411.4555. ↗
  289. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. 2015. find
  290. Nguyen Vo and Kyumin Lee. Learning from fact-checkers: Analysis and generation of fact-checking language. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, pp. 335–344, New York, NY, USA, 2019. Association for Computing Mach… — [doi:10.1145/3331184.3331248](https://doi.org/10.1145/3331184.3331248) find
  291. Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. 2020a. find
  292. Hongmin Wang. Revisiting challenges in data-to-text generation with fact grounding. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 311–322, Tokyo, Japan, October–November 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-8639. URL https://… find
  293. Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. Towards faithful neural table-to-text generation with content-matching constraints. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020b. doi: 10.1…
  294. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. 2019.
  295. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),…
  296. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. Challenges in data-to-document generation, 2017.
  297. Billy T. M. Wong and Chunyu Kit. Extending machine translation evaluation metrics with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1060–1068, 2019. find
  298. Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In ArXiv, volume abs/2001.04063, 2020. — [arXiv:2001.04063](https://arxiv.org/abs/2001.04063) find
  299. Qian Yang, Rebecca J. Passonneau, and Gerard de Melo. Peak: Pyramid evaluation via automated knowledge extraction. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016. find
  300. Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and-write: Towards better automatic storytelling. In CoRR, volume abs/1811.05701, 2018. URL http://arxiv.org/abs/1811.05701. ↗
  301. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. Dependency-based discourse parser for single-document summarization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1834– 1839, Doha, Qatar, October 2014. Association for Computat…
  302. R. Michael Young. Using grice’s maxim of quantity to select the content of plan descriptions. In Artif. Intell., volume 115, pp. 215–256, 1999. find
  303. Hui Yu, Xiaofeng Wu, Jun Xie, Wenbin Jiang, Qun Liu, and Shouxun Lin. RED: A reference dependency based MT evaluation metric. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 2042–2051, Dublin, Ireland, August 2014. Dublin City Univ…
  304. Hui Yu, Xiaofeng Wu, Wenbin Jiang, Qun Liu, and ShouXun Lin. An automatic machine translation evaluation metric based on dependency parsing model. 08 2015. find
  305. Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 15–25. Association for Co… find
  306. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS, 2019.
  307. Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. Evaluating machines by their real-world language use. In ArXiv, volume abs/2004.03607, 2020. — [arXiv:2004.03607](https://arxiv.org/abs/2004.03607) find
  308. Jacob Zerr. Question generation using part of speech information. FINAL REPORT FOR REU PROGRAM AT UCCS, 2014. URL http://cs.uccs.edu/~jkalita/work/reu/REU2014/FinalPapers/Zerr.pdf. ↗
  309. Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), Online, April 2021. Association for Computational Linguistics. URL https://www.aclweb.… ↗
  310. Qiuyun Zhang, Bin Guo, Hao Wang, Yunji Liang, Shaoyang Hao, and Zhiwen Yu. Ai-powered text generation for harmonious human-machine interaction: Current state and future directions. In CoRR, volume abs/1905.01984, 2019a. URL http://arxiv.org/abs/1905.01984. — [arXiv:1905.01984](https://arxiv.org/abs/1905.01984) ↗
  311. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020a. URL https://openreview.net/forum?id=SkeHuCVFDr. ↗
  312. Ying Zhang, Stephan Vogel, and Alex Waibel. Interpreting bleu/nist scores: How much improvement do we need to have a better system. In In Proceedings of Proceedings of Language Resources and Evaluation (LREC-2004, pp. 2051–2054, 2004. find
  313. Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, and Curtis P. Langlotz. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In arXiv 1911.02541, 2019b.
  314. Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5108–5120, Online, July 2020b.…
  315. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In EMNLP, 2019.
  316. Giulio Zhou and Gerasimos Lampouras. WebNLG challenge 2020: Language agnostic delexicalisation for multilingual RDF-to-text generation. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pp. 186–191, Dublin, Ireland (Virtual), 12 2020. As… find
  317. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. The design and implementation of xiaoice, an empathetic social chatbot. In Computational Linguistics, volume 46. MIT Press, 2020.
  318. Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models, 2020. find
  319. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. Boosting factual correctness of abstractive summarization, 2020. find
  320. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In CoRR, volume abs/1802.01886, 2018. URL http://arxiv.org/abs/1802.01886. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/celikyilmaz-2020-evaluation-text-generation-survey/