Elamur Research
☰
home / papers / 2017 / Creating Training Corpora for NLG Micro-Planners

Creating Training Corpora for NLG Micro-Planners

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1017.pdf

Summary

The paper addresses the lack of training data for NLG micro-planners. These are data-to-text systems that must jointly handle lexicalisation, aggregation, surface realisation, sentence segmentation and referring expression generation. Existing resources fall short for different reasons. Domain-specific corpora yield stereotyped text, and expert-annotated benchmarks (SR'11, AMR Corpus) mostly reduce the task to surface realisation. Crowdsourced data such as Wen et al. (2016) uses artificial, shallow inputs. The authors propose a semi-automatic framework that applies to any large categorised knowledge base. A content selection step extracts varied, coherent sets of RDF triples, including deeper tree shapes that call for richer syntax. Crowdworkers then write texts for these inputs, and majority-vote validation discards mismatched pairs. Applied to DBpedia, the framework produced a dataset that is less than half the size of Wen et al.'s but more diverse in both inputs and texts. A sequence-to-sequence model also performs worse on it. The resource therefore offers a harder benchmark and supports training verbalisers for real KBs.

Key points

  • The first released WebNLG dataset, built for a micro-planning shared task, has 21,855 data/text pairs over 8,372 distinct inputs. The inputs cover 9 DBpedia categories, including Astronaut, University, Monument, Airport and SportsTeam. It is licensed under CC BY-NC-SA 4.0.
  • Compared with Wen et al.'s RNNLG data, WebNLG texts have a higher corrected type-token ratio (CTTR) and higher lexical sophistication (LS). Both measures were computed with Lu's (2012) Lexical Complexity Analyzer.
  • In the TensorFlow seq2seq experiment, WebNLG gave higher perplexity than RNNLG in both delexicalisation modes (27.41 and 25.39 vs. 17.42 and 23.93). It also gave lower BLEU (0.19 and 0.10 vs. 0.26 and 0.27).
  • The ratio of inputs to input patterns is about five times lower in WebNLG, which makes learning harder but reduces overfitting. WebNLG also has roughly twice as many texts per input, which helps with learning paraphrases.
  • The authors excluded the Lebret et al. (2016) Wikipedia infobox dataset. In a manual check of 50 random pairs, none had text that matched its data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

144 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1017

@inproceedings{gardent2017creating,
  title = {{Creating Training Corpora for NLG Micro-Planners}},
  author = {Gardent, Claire and Shimorina, Anastasia and Narayan, Shashi and Perez-Beltrachini, Laura},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1017},
  url = {https://doi.org/10.18653/v1/p17-1017}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2012. Abstract meaning representation (AMR) 1.0 specification. In Proceedings of EMNLP. find
  2. Eva Banik, Claire Gardent, and Eric Kow. 2013. The KBGen challenge. In Proceedings of ENLG. find
  3. Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The 14 The test data for the W EB NLG challenge will be released on August 18th, 2017 and preliminary results will be presented and discussed at INLG 2017, https:// eventos.citius.usc.es/inlg2017/index. 13 S… find
  4. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of EMNLP. first surface realisation shared task: Overview and evaluation results. In Proceed… find
  5. David L Chen and Raymond J Mooney. 2008. Learning to sportscast: A test of grounded language acquisition. In Proceedings of ICML. find
  6. Gerasimos Lampouras and Andreas Vlachos. 2016. Imitation learning for language generation from unaligned data. In Proceedings of COLING. find
  7. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural Text Generation from Structured Data with Application to the Biography Domain. In Proceedings of EMNLP. find our card
  8. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning Semantic Correspondences with Less Supervision. In Proceedings of ACL-IJCNLP. find
  9. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal 96(2):190–208. find
  10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL:System Demonstrations. find our card
  11. Pablo N Mendes, Max Jakob, and Christian Bizer. 2012. DBpedia: A Multilingual Cross-domain Knowledge Base. In Proceedings of LREC. find
  12. Jekaterina Novikova and Verena Rieser. 2016. The aNALoGuE challenge: Non aligned language generation. In Proceedings of INLG. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  14. Laura Perez-Beltrachini, Rania Mohamed Sayed, and Claire Gardent. 2016. Building RDF content for Data-to-Text generation. In Proceedings of COL-ING. find
  15. Adwait Ratnaparkhi. 2000. Trainable methods for surface natural language generation. In Proceedings of NAACL. find
  16. Andreas Stolcke. 2002. SRILM – An extensible language modeling toolkit. In Proceedings of ICSLP. find
  17. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Proceedings of NIPS. find
  18. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of NAACL-HLT. find