Elamur Research⌕
☰
home / papers / 2020 / Text-to-Text Pre-Training for Data-to-Text Tasks

Text-to-Text Pre-Training for Data-to-Text Tasks

Paper · 2020 · Mihir Kale and Abhinav Rastogi

Summary

The paper asks whether general-purpose text-to-text pre-training can replace specialised architectures for data-to-text generation, where structured input such as tables, dialogue acts or RDF-style triples must be turned into fluent text. The authors flatten each structured input into a plain string and fine-tune the released T5 encoder-decoder checkpoints, from Small (60M parameters) to 3B, for a few thousand steps. They test on three English benchmarks with different input types: WebNLG (graphs of triples), ToTTo (Wikipedia table cells with metadata) and MultiWoz (dialogue meaning representations). This plain end-to-end setup outperforms pipelined systems, graph-network encoders and models built on BERT or GPT-2. The largest gains appear on out-of-domain test splits, and human raters confirm them. The work shows that strong pre-training can make complex task-specific designs unnecessary. It also gives a simple, reproducible baseline for future data-to-text research, which the authors hope to extend to other languages, including low-resource ones.

Key points

  • Fine-tuning setup: constant learning rate of 0.001, inputs capped at 512 tokens, greedy decoding, and checkpoints selected by dev-set BLEU. The model is trained for 5K steps on WebNLG and 10K steps on ToTTo.
  • On WebNLG, T5-Large beats the previous best system DualEnc by 4.3 BLEU overall and by about 14 BLEU on the unseen-domain split. In human evaluation on unseen inputs, T5-Large reaches 90% accuracy and 2.33 naturalness, while DualEnc reaches 66% and 1.99.
  • On the ToTTo test set, T5-3B scores 49.5 BLEU and 58.4 PARENT, improving on BERT-to-BERT by 5.5 BLEU and 5.8 PARENT. The margin grows to 6.6 BLEU and 7.5 PARENT on the out-of-domain Non-Overlap half.
  • On MultiWoz, every T5 size beats SC-GPT2 by 4–5 BLEU without any in-domain pre-training. Slot Error Rate is slightly worse, partly because SC-GPT2 reranks five candidate outputs by SER.
  • The benefit of larger models depends on the dataset. Size barely matters on the large, homogeneous MultiWoz, but moving from T5-Small to T5-Base adds about 10 BLEU on WebNLG's unseen split.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksText generation · 214 works

Cite

Cite as Mihir Kale and Abhinav Rastogi. 2020. Text-to-Text Pre-Training for Data-to-Text Tasks. arXiv:2005.10433.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.10433 · arXiv 2005.10433

@misc{kale2020text,
  title = {{Text-to-Text Pre-Training for Data-to-Text Tasks}},
  author = {Mihir Kale and Abhinav Rastogi},
  year = {2020},
  doi = {10.48550/arxiv.2005.10433},
  eprint = {2005.10433},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.10433}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P…
  2. Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019a. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696–3709. find
  3. Zhiyu Chen, Harini Eavani, Yinyin Liu, and William Yang Wang. 2019b. Few-shot nlg with pre-trained language model. arXiv preprint arXiv:1904.09521. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  5. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884–4895.
  6. Bayu Distiawan, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. Gtr-lstm: A triple encoder for sentence generation from rdf data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1627–1637. find
  7. Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Jo… find
  8. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133.
  9. Eli Goldberg, Norbert Driedger, and Richard I Kittredge. 1994. Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45– 53. find
  10. Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of the 21st annual meeting on Association for Computational Linguistics, pages 145–150. Association for Computational Linguistics.
  11. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231. Association for Computational Linguistics.
  12. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  13. Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  14. Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep graph convolutional encoders for structured data to text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 1–9. find
  15. Kathleen R McKeown. 1985. Text generation: using discourse strategies and focus constraints to generate natural language text. find
  16. Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long…
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
  18. Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373.
  19. Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002.12328.
  20. Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6908–6915. find
  21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  23. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press. find
  24. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2019. Leveraging pre-trained checkpoints for sequence generation tasks. arXiv preprint arXiv:1907.12461.
  25. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017a. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Compu… find
  26. Abigail See, Peter J Liu, and Christopher D Manning. 2017b. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083. find
  27. Somayajulu Sripada, Ehud Reiter, and Ian Davy. Sumtime-mousam: Configurable marine weather forecast generator. find
  28. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
  29. Chao Zhao, Marilyn Walker, and Snigdha Chaturvedi. 2020. Bridging the structural gap between encoding and decoding for data-to-text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kale-2020-text-text-pre-training-data/