Elamur Research
☰
home / papers / 2017 / The E2E Dataset: New Challenges For End-to-End Generation

The E2E Dataset: New Challenges For End-to-End Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-5525.pdf

Summary

End-to-end NLG systems handle sentence planning and surface realisation together, learning directly from unaligned pairs of meaning representations (MRs) and texts. Until this work, they had been trained mostly on small, delexicalised corpora such as BAGEL and SF Restaurants, which produced fairly template-like output. The authors introduce the E2E dataset, about 50k crowdsourced pairings of dialogue-act-style MRs with human-written restaurant descriptions. The data was gathered on CrowdFlower, and workers were shown pictures instead of textual MRs as prompts, which encourages more natural phrasing. The authors compare the corpus with BAGEL and SFRest using the analysis framework of Perez-Beltrachini and Gardent (2017). E2E turns out to be much larger, lexically richer and syntactically more varied, with longer, multi-sentence references. Workers also often skip or add information, so a generator must learn content selection as well as wording. A TGen seq2seq baseline reaches BLEU scores similar to its results on the older corpora. This suggests that the extra size and multiple references compensate for the harder data. The resource was released for the E2E NLG Challenge as a shared, more demanding benchmark.

Key points

  • The E2E dataset has 50,602 instances and 5,751 unique MRs, with on average 8.1 references per MR (range 2–16). Each MR has 3–8 slots, and the data is split 76.5/8.5/15 into train/validation/test with no MR overlap between sets.
  • Lexical richness is higher than in the comparison corpora:
    • MSTTR is 0.75 for E2E, against 0.62 for SFRest and 0.41 for BAGEL.
    • Lexical sophistication (LS) is 0.57, against 0.43 and 0.42.
    • Half of the bigrams and 61% of the trigrams occur only once.
  • References run up to 6 sentences and average 20.1 words, against 8.53 for SFRest and 11.54 for BAGEL. E2E also has the largest shares of coordinated sentences (14%) and of the most complex D-Level sentences (16%).
  • In a manual check of 50 MR–reference pairs, 22% omitted content and 18% added content, so about 40% do not match the MR exactly. Both SFRest and BAGEL show near-complete coverage.
  • The TGen baseline, with delexicalised name/near and evaluated on the dev set, scored:
    • BLEU 0.6925
    • NIST 8.4781
    • METEOR 0.4703
    • ROUGE-L 0.7257
    • CIDEr 2.3987

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

241 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 229 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-5525 · arXiv 1706.09254

@inproceedings{novikova2017e2e,
  title = {{The E2E Dataset: New Challenges For End-to-End Generation}},
  author = {Novikova, Jekaterina and Dušek, Ondřej and Rieser, Verena},
  booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue},
  year = {2017},
  doi = {10.18653/v1/w17-5525},
  eprint = {1706.09254},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w17-5525}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. San Diego, CA, USA. ArXiv:1409.0473. http://arxiv.org/abs/1409.0473. ↗ our card
  2. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 228– 231. http:/… find our card
  3. David L. Chen and Raymond J. Mooney. 2008. Learning to sportscast: A test of grounded language acquisition. In Proceedings of the 25th international conference on Machine learning (ICML). Helsinki, Finland, pages 128–135. http://dl.acm.org/citation.cfm?id=1390173. ↗
  4. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-The training and development parts of our dataset can be downloaded from http://www.macs.hw.ac.uk/ InteractionLab/E2E/. information presentation for statistical dial… ↗
  5. Xiaofei Lu. 2009. Automatic measurement of syntactic complexity in child language acquisition. International Journal of Corpus Linguistics 14(1):3–28. http://doi.org/10.1075/ijcl.14.1.02lu. — [doi:10.1075/ijcl.14.1.02lu](https://doi.org/10.1075/ijcl.14.1.02lu) ↗
  6. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2016. Natural language generation in dialogue using lexicalized and delexicalized data. CoRR abs/1606.03632. http://arxiv.org/abs/1606.03632. — [arXiv:1606.03632](https://arxiv.org/abs/1606.03632) ↗
  7. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal 96(2):190–208. http://doi.org/10.1111/j.1540-4781.2011.01232 1.x. — [doi:10.1111/j.1540-4781.2011.01232](https://doi.org/10.1111/j.1540-4781.2011.01232) ↗
  8. Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentations in spoken dialog systems. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics. Barcelona, Spain, pages 79–86. http://aclweb.org/anthology/P04-1011. ↗
  9. Xiaofei Lu. 2014. Computational methods for corpus annotation and analysis. Springer. http://doi.org/10.1007/978-94-017-8645-4. — [doi:10.1007/978-94-017-8645-4](https://doi.org/10.1007/978-94-017-8645-4) ↗
  10. François Mairesse, Milica Gašić, Filip Jurčı́ček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Upp… find
  11. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, pages 4566–4575. https://doi.org/10.1109/CVPR.2015.7299087. ↗ our card
  12. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San… find
  13. Marilyn A Walker, Stephen J Whittaker, Amanda Stent, Preetam Maloor, Johanna Moore, Michael Johnston, and Gunaranjan Vasireddy. 2004. Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science 28(5):811–840. https://doi.org/10.1016/j.cogsci.2004.06.002. — [doi:10.1016/j.cogsci.2004.06.002](https://doi.org/10.1016/j.cogsci.2004.06.002) ↗
  14. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, and Steve J. Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Co… find
  15. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation Conference. Edinburgh, UK, pages 265–273. arXiv:1608.00339. http://aclweb.org/anthology/W16-2302. ↗ our card
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, PA, USA, pages 311–3… find our card
  17. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal… find
  18. Laura Perez-Beltrachini and Claire Gardent. 2017. Analysing data-to-text generation benchmarks. In Proceedings of the 10th International Natural Language Generation Conference. Santiago de Compostela, Spain. http://arxiv.org/abs/1705.03802. ↗
  19. Verena Rieser and Oliver Lemon. 2009. Natural language generation as planning under uncertainty for spoken dialogue systems. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL). Athens, Greece, pages 683–691. http://aclweb.org/anthology/E09-1078. ↗
  20. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incremental planning under uncertainty: Adaptive find