Elamur Research
☰
home / papers / 2020 / ToTTo: A Controlled Table-To-Text Generation Dataset

ToTTo: A Controlled Table-To-Text Generation Dataset

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.89.pdf

Summary

ToTTo is an open-domain English dataset for table-to-text generation from Google Research. It targets a known weakness of neural generators: they often hallucinate content that the source data does not support. Each input is a Wikipedia table with its metadata and a set of highlighted cells, and the model must write one sentence describing those cells. Highlighting tells the model which part of a possibly large table to describe without fixing an exact meaning representation, so the task still requires content planning and some reasoning. The targets are not written from scratch. Wikipedia sentences that overlap with a table were collected using heuristics, and annotators then edited each one step by step. They marked the cells that support it, removed unsupported phrases, and resolved pronouns and context so the sentence stands on its own. The result keeps the variety of natural text while staying faithful to the table. Strong baselines such as BERT-to-BERT produce fluent sentences but still fall short of the references on faithfulness and coverage. This makes ToTTo a useful benchmark for high-precision generation and for developing better evaluation metrics.

Key points

  • More than 120,000 training examples drawn from many kinds of Wikipedia tables, including irregular ones with merged cells. Infobox tables were excluded so the data does not overlap with Wikibio.
  • Annotation pipeline: highlight the supporting cells, delete unsupported phrases, decontextualize the sentence, then do a final grammar fix.
  • In human evaluation, both baselines covered less content than the references, and their faithfulness was significantly lower (p < 0.001).
  • Typical errors include hallucinated phrases, mistakes on rare topics, and failures at numerical or structural reasoning such as counting rows. BLEU often misses these errors.
  • The dataset is released at github.com/google-research-datasets/totto, together with a leaderboard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

238 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 223 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.89

@inproceedings{parikh2020totto,
  title = {{ToTTo: A Controlled Table-To-Text Generation Dataset}},
  author = {Parikh, Ankur and Wang, Xuezhi and Gehrmann, Sebastian and Faruqui, Manaal and Dhingra, Bhuwan and Yang, Diyi and Das, Dipanjan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.89},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.89}
}

References

41 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):21. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proc. of EMNLP. find our card
  3. Eva Banik, Claire Gardent, and Eric Kow. 2013. The kbgen challenge. In Proc. of European Workshop on NLG. find
  4. David L Chen and Raymond J Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In Proc. of ICML. find
  5. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020. Logical natural language generation from open-domain tables. In Proc. of ACL. find
  6. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. In Proc. of ICLR. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL. find
  8. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proc. of ACL. find our card
  9. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL. find our card
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. find
  11. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378. find
  12. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a. Creating training corpora for NLG micro-planning. In Proc. of ACL. find
  13. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b. The WebNLG challenge: Generating text from RDF data. In Proc. of INLG. find our card
  14. Sebastian Gehrmann, Falcon Z Dai, Henry Elder, and Alexander M Rush. 2018. End-to-end content and plan selection for data-to-text generation. In INLG. find
  15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL. find
  16. Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. TACL, 6:437–450. find our card
  17. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. find
  18. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proc. of WMT. find
  19. Flip Korn, Xuezhi Wang, You Wu, and Cong Yu. 2019. Automatically generating interesting facts from wikipedia tables. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD 19, page 349361, New York, NY, USA. Association for Computing Machinery. find
  20. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proc. of EMNLP. find our card
  21. Karen Kukich. 1983. Design of a knowledge-based report generator. In Proc. of ACL. find
  22. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  23. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proc. of EMNLP. find our card
  24. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. In Open Review. find
  25. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proc. of ACL. find
  26. Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 57–66. find
  27. Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Proc. of AAAI. find
  28. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90. find our card
  29. Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press. find
  30. Kathleen McKeown. 1992. bridge University Press. find
  31. Text generation. Cam-Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proc. of SIGDIAL. find our card
  32. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing nlg data: Pictures elicit better data. In Proc. of INLG. find
  33. Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proc. of ACL. find
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL. find our card
  35. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proc. of ACL. find
  36. Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. In Proc. of NAACL. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS. find
  38. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proc. of ICML Deep Learning Workshop. find our card
  39. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proc. of EMNLP. find
  40. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. In Proc. of EMNLP. find our card
  41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proc. of ICCV. find our card