Elamur Research
☰
home / papers / 2018 / Texygen: A Benchmarking Platform for Text Generation Models

Texygen: A Benchmarking Platform for Text Generation Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1802.01886

Summary

Comparing models for open-domain text generation, especially GAN-based ones, is unreliable for three reasons: no single metric captures model quality, many authors don't release code, and gains in output quality often just come from giving up diversity. Texygen addresses this as an open-source benchmarking platform built on TensorFlow. It re-implements a set of baseline generators and puts several automatic metrics behind one shared API. The metrics cover three things: how closely outputs resemble real text, how well they fit the data in likelihood terms, and how diverse they are. The authors run the baselines in two settings. The first is synthetic: a randomly initialised LSTM "oracle" produces the training data. The second is real: captions from COCO. The runs show trade-offs that a single score would hide. LeakGAN converges fastest and keeps its embedding similarity after adversarial training starts, yet its outputs are also the least diverse. The plain MLE model gives the most varied text. A shared, reproducible setup like this makes later work on text generation easier to compare.

Key points

  • Baselines: vanilla MLE language model, SeqGAN, MaliGAN, RankGAN, TextGAN, GSGAN (Gumbel-Softmax) and LeakGAN. The code is at github.com/geek-ai/Texygen. Oracles can be LSTM-, GRU- or SRU-based.
  • Metrics: BLEU, NLL-oracle, and three that the authors propose here: EmbSim (compares word-embedding similarity matrices built with skip-gram), NLL-test (fit to real test data) and Self-BLEU (how much generated sentences overlap with each other, used as a diversity proxy where higher means more mode collapse).
  • Setup: The synthetic data has a 5,000-word vocabulary and 10,000 oracle sentences of 20 tokens each. The real data is 20,000 COCO captions split evenly into training and test sets. Training uses 80 epochs of generator pretraining, then 80 of discriminator pretraining, then 100 adversarial epochs.
  • Self-BLEU-5: LeakGAN 0.780 is the highest (least diverse); MLE 0.408 is the lowest. On BLEU-5 against training data, TextGAN (0.523) beats LeakGAN (0.470).
  • Excluded or unhelpful: GSGAN produced no meaningful sentences on the real data. A CFG-based grammar metric could not tell the models apart and tended to favour mode-collapsed models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

338 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 327 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.01886 · arXiv 1802.01886

@misc{zhu2018texygen,
  title = {{Texygen: A Benchmarking Platform for Text Generation Models}},
  author = {Zhu, Yaoming and Lu, Sidi and Zheng, Lei and Guo, Jiaxian and Zhang, Weinan and Wang, Jun and Yu, Yong},
  year = {2018},
  doi = {10.48550/arxiv.1802.01886},
  eprint = {1802.01886},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.01886}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi et al. 2016. TensorFlow: A System for Large-Scale Machine Learning.. In OSDI, Vol. 16. 265–283. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014). find our card
  3. Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. arXiv preprint arXiv:1702.07983 (2017). — [arXiv:1702.07983](https://arxiv.org/abs/1702.07983) find
  4. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680. find
  5. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Long Text Generation via Adversarial Training with Leaked Information. arXiv preprint arXiv:1709.08624. — [arXiv:1709.08624](https://arxiv.org/abs/1709.08624) find
  6. Ferenc Huszár. 2015. How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? arXiv preprint arXiv:1511.05101 (2015). — [arXiv:1511.05101](https://arxiv.org/abs/1511.05101) find
  7. Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity - a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America 62, S1 (1977), S63–S63. find
  8. Matt J Kusner and José Miguel Hernández-Lobato. 2016. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051 (2016). — [arXiv:1611.04051](https://arxiv.org/abs/1611.04051) find
  9. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547 (2017). find
  10. Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial Ranking for Language Generation. arXiv preprint arXiv:1705.11001 (2017). — [arXiv:1705.11001](https://arxiv.org/abs/1705.11001) find
  11. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. 311–318. find our card
  13. Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In SIGIR. ACM, 515–524. find
  14. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, 3-4 (1992), 229–256. find
  15. Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv preprint arXiv:1502.03044. find our card
  16. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.. In AAAI. 2852–2858. find
  17. Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial Feature Matching for Text Generation. arXiv preprint arXiv:1706.03850 (2017). *Table 2: BLEU score on test data SeqGAN MaliGAN RankGAN BLEU-2 0.745 0.673 0.743 BLEU-3 0.498 0.432 0.467 BLEU-4… — [arXiv:1706.03850](https://arxiv.org/abs/1706.03850) find