Elamur Research⌕
☰
home / papers / 2019 / Semantic Noise Matters for Neural Natural Language Generation

Semantic Noise Matters for Neural Natural Language Generation

Summary

Neural generators that turn meaning representations (MRs) into text often produce output that does not match their input. This paper asks how much of that problem comes from noisy crowdsourced training data. The authors used the E2E restaurant dataset, where about 40% of instances leave out or add information. Rather than rewriting the human references, they automatically re-annotated each MR to match its text, using a regular-expression slot matching script based on Reed et al. (2018). They also removed training and development items that overlapped with the test set. They then trained TGen, in three reranking variants, and SC-LSTM on both the original and the cleaned data. Each model was evaluated on both test sets using word-overlap metrics, slot error rate (SER) and a manual error analysis. Training on cleaned data cut SER by up to 97% and lowered fluency only slightly. This effect was far larger than the benefit of a semantic reranker alone. The dominant error type was omission, not hallucination. The study shows that cleaning data is a cheap and effective way to make neural NLG more faithful to its input.

Key points

  • Cleaned E2E corpus: The training set grew from 4,862 to 8,362 distinct MRs. Its references dropped from 42,061 to 33,525 after removing items that overlapped with the test set. Data and scripts are released at github.com/tuetschek/e2e-cleaning.
  • Script accuracy: On 200 manually checked training instances, the slot matching script had a 4.2% SER. On 400 system outputs, it matched 99.5% of instances correctly.
  • SER gains for TGen (original test set): With the default reranker, SER fell from 4.27% to 0.12%, about a 97% reduction. Without the reranker (TGen−), it fell from 15.94% to 0.97%, about 94%. Word-overlap scores dipped slightly, except NIST.
  • Model comparison: SC-LSTM performed much worse than TGen, with SER above 30%. The authors attribute this to how much delexicalisation it needs.
  • Error types: Missing slots were the main error, and cleaning the missing-slot noise gave the biggest improvement. The cleaned test set proved harder because its MRs are more diverse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-8652

@inproceedings{dusek2019semantic,
  title = {{Semantic Noise Matters for Neural Natural Language Generation}},
  author = {Dušek, Ondřej and Howcroft, David M. and Rieser, Verena},
  booktitle = {Proceedings of the 12th International Conference on Natural Language Generation},
  year = {2019},
  doi = {10.18653/v1/w19-8652},
  url = {https://doi.org/10.18653/v1/w19-8652}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations (ICLR2015), San Diego, CA, USA. arXiv:1409.0473. find our card
  2. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145, San Diego, CA, USA. find
  3. Ondřej Dušek and Filip Jurčı́ček. 2016. Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–51, Berlin, Germany. arXiv:1606.05491. find
  4. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG Challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 322–328, Tilburg, The Netherlands. arXiv:1810.01170. find
  5. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, 59:123–156. arXiv:1901.07931. https://ehudreiter.com/ — [arXiv:1901.07931](https://arxiv.org/abs/1901.07931) ↗
  6. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines. arXiv:1907.01669. find
  7. Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019. A simple recipe towards reducing hallucination in neural surface realisation. In Proceedings of the 57th Conference of the Association for Computational Linguistics Volume 1: Long Papers, pages 2673–2679, Florence, Italy. find
  8. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of Neural Models Make Interpretations Difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. find
  9. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG Data: Pictures Elicit Better Data. In The 9th International Natural Language Generation conference INLG, Edinburgh, Scotland, UK. arXiv: 1608.00339. find
  10. David M. Howcroft, Dietrich Klakow, and Vera Demberg. 2017. The Extended SPaRKy Restaurant Corpus: Designing a Corpus with Variable Information Density. In Proceedings of Interspeech 2017, pages 3757–3761, Stockholm, Sweden. find
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA. find our card
  12. Juraj Juraska, Panagiotis Karagiannis, Kevin K. Bowden, and Marilyn A. Walker. 2018. A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  13. Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver, Canada. find
  14. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic. find our card
  15. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation. OpenReview. find
  16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, pages 74–81, Barcelona, Spain. find
  17. F. Mairesse, M. Gašić, F. Jurčı́ček, S. Keizer, B. Thomson, K. Yu, and S. Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, page 1552–1561, Uppsala, Swe… find
  18. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… find our card
  19. Lena Reed, Shereen Oraby, and Marilyn Walker. 2018. Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring? In Proceedings of the 11th International Conference on Natural Language Generation, pages 284–295, Tilburg, The Netherlands. arXiv:1809.03015. find
  20. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, Brussels, Belgium. find our card
  21. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, Boston, MA, USA. find our card
  22. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computationa… find
  23. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, L… find our card
  24. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2017. Challenges in Data-to-Document Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2243–2253, Copenhagen, Denmark. arXiv:1707.08052. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/