Elamur Research
☰
home / papers / 2017 / Learning to Generate Reviews and Discovering Sentiment

Learning to Generate Reviews and Discovering Sentiment

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1704.01444

Summary

Unsupervised sentence representations such as skip-thought vectors still trail supervised models. The authors suspect two causes: training text that differs from the target tasks, and models too small to keep fine-grained detail. To test this, they trained a byte-level language model on the Amazon product review corpus of McAuley et al. (2015). The model is a single-layer multiplicative LSTM (mLSTM) with 4096 units that predicts each next UTF-8 byte. Its final cell states then serve as features for a logistic regression classifier.

The resulting representation is especially strong for sentiment. It sets a new state of the art on the binary Stanford Sentiment Treebank, MR and CR, and stays competitive when given only a few labeled examples. Sentiment is largely captured by a single hidden unit. Clamping that unit to a positive or negative value steers generated reviews toward that polarity. On out-of-domain tasks such as Yelp and SICK the advantage mostly disappears. This shows how strongly learned features depend on the training distribution. It also shows that an unmodified language modelling objective can produce useful, interpretable representations.

Key points

  • Training data: a de-duplicated Amazon review corpus of over 82 million reviews (1996–2014), about 38 billion bytes, split into 1000 shards.
  • Training setup: one epoch, about 1 million updates with Adam and weight normalization, run on 4 Pascal Titan X GPUs for roughly a month. The model reached 1.12 bits per byte.
  • Text classification (Table 1): the byte mLSTM scored 86.9 on MR and 91.4 on CR, both new bests. It had no clear edge on SUBJ (94.6) or MPQA (88.5).
  • Sentiment unit: one unit encodes sentiment. Fixing it to −1 or 1 during sampling yields negative or positive reviews. Negative samples sometimes include positive sentences, possibly because five-star reviews outnumber one-star reviews more than 5 to 1 in the corpus.
  • Other tasks: the model is competitive on the Microsoft Research Paraphrase Corpus but weak on SICK semantic relatedness. The authors attribute this to SICK being out of domain for a model trained on product reviews.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 124 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1704.01444 · arXiv 1704.01444

@misc{radford2017learning,
  title = {{Learning to Generate Reviews and Discovering Sentiment}},
  author = {Radford, Alec and Jozefowicz, Rafal and Sutskever, Ilya},
  year = {2017},
  doi = {10.48550/arxiv.1704.01444},
  eprint = {1704.01444},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1704.01444}
}

References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  2. Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013. find our card
  3. Blei, David M, Ng, Andrew Y, and Jordan, Michael I. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003. find our card
  4. Chen, Danqi and Manning, Christopher D. A fast and accurate dependency parser using neural networks. In EMNLP, pp. 740–750, 2014. find our card
  5. Coates, Adam, Lee, Honglak, and Ng, Andrew Y. An analysis of single-layer networks in unsupervised feature learning. Ann Arbor, 1001(48109):2, 2010. find
  6. Collobert, Ronan, Weston, Jason, Bottou, Léon, Karlen, Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537, 2011. find our card
  7. Dai, Andrew M and Le, Quoc V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pp. 3079–3087, 2015. find
  8. Dieng, Adji B, Wang, Chong, Gao, Jianfeng, and Paisley, John. Topicrnn: A recurrent neural network with long-range semantic dependency. arXiv preprint arXiv:1611.01702, 2016. — [arXiv:1611.01702](https://arxiv.org/abs/1611.01702) find
  9. Dolan, Bill, Quirk, Chris, and Brockett, Chris. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, pp. 350. Association for Computational Linguistics, 2004. find our card
  10. Dong, Li, Huang, Shaohan, Wei, Furu, Lapata, Mirella, Zhou, Ming, and Ke, Xu. Learning to generate product reviews from attributes. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pp. 623–632. Association for Computational Linguistics,… find
  11. Goodfellow, Ian, Bengio, Yoshua, and Courville, Aaron. Deep learning. 2016. find
  12. Hill, Felix, Cho, Kyunghyun, and Korhonen, Anna. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016. find our card
  13. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processin… find
  14. Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006. find
  15. Hu, Minqing and Liu, Bing. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168–177. ACM, 2004. find
  16. Hu, Zhiting, Yang, Zichao, Liang, Xiaodan, Salakhutdinov, Ruslan, and Xing, Eric P. Controllable text generation. arXiv preprint arXiv:1703.00955, 2017. find our card
  17. Huang, Fu Jie, Boureau, Y-Lan, LeCun, Yann, et al. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. find
  18. Hutter, Marcus. The human knowledge compression contest. 2006. URL http://prize. hutter1. net, 2006. find
  19. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  20. Karpathy, Andrej, Johnson, Justin, and Fei-Fei, Li. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015. find our card
  21. Kim, Yoon. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. find our card
  22. Kingma, Diederik and Ba, Jimmy. method for stochastic optimization. arXiv:1412.6980, 2014. find
  23. Adam: A arXiv preprint Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. find
  24. Krause, Ben, Lu, Liang, Murray, Iain, and Renals, Steve. Multiplicative lstm for sequence modelling. arXiv preprint arXiv:1609.07959, 2016. — [arXiv:1609.07959](https://arxiv.org/abs/1609.07959) find
  25. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012. find
  26. Kumar, Ankit, Irsoy, Ozan, Su, Jonathan, Bradbury, James, English, Robert, Pierce, Brian, Ondruska, Peter, Gulrajani, Ishaan, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285, 2015. find our card
  27. Le, Quoc V. Building high-level features using large scale unsupervised learning. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 8595–8598. find
  28. Li, Jiwei, Monroe, Will, and Jurafsky, Dan. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  29. Looks, Moshe, Herreshoff, Marcello, Hutchins, DeLesley, and Norvig, Peter. Deep learning with dynamic computation graphs. arXiv preprint arXiv:1702.02181, 2017. — [arXiv:1702.02181](https://arxiv.org/abs/1702.02181) find
  30. Maas, Andrew L, Daly, Raymond E, Pham, Peter T, Huang, Dan, Ng, Andrew Y, and Potts, Christopher. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pp. 142–150. Association f… find our card
  31. Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Re-examining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 182–190. Association f… find
  32. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. find our card
  33. Marelli, Marco, Bentivogli, Luisa, Baroni, Marco, Bernardi, Raffaella, Menini, Stefano, and Zamparelli, Roberto. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval-2014, 2014. find our card
  34. McAuley, Julian, Pandey, Rahul, and Leskovec, Jure. Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. ACM, 2015. find
  35. Mesnil, Grégoire, Mikolov, Tomas, Ranzato, Marc’Aurelio, and Bengio, Yoshua. Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews. arXiv preprint arXiv:1412.5335, 2014. — [arXiv:1412.5335](https://arxiv.org/abs/1412.5335) find
  36. Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey. Linguistic regularities in continuous space word representations. 2013. find
  37. Miyato, Takeru, Dai, Andrew M, and Goodfellow, Ian. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725, 2016. find our card
  38. Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315, 2016. find
  39. Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, pp. 78. ACM, 2004. find
  40. Olshausen, Bruno A and Field, David J. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. find
  41. Oquab, Maxime, Bottou, Leon, Laptev, Ivan, and Sivic, Josef. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717–1724, 2014. find
  42. Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics, pp. 271. Association for Computational Linguistics, 2004. find
  43. Pang, Bo and Lee, Lillian. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pp. 115– 124. Association for Computational Linguistics, 2005. find
  44. Pennington, Jeffrey, Socher, Richard, and Manning, Christopher D. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532–1543, 2014. find
  45. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901–901, 2016. find
  46. Socher, Richard, Perelygin, Alex, Wu, Jean Y, Chuang, Jason, Manning, Christopher D, Ng, Andrew Y, Potts, Christopher, et al. Recursive deep models for semantic compositionality over a sentiment treebank. Citeseer, 2013. find
  47. Tai, Kai Sheng, Socher, Richard, and Manning, Christopher D. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075, 2015. find
  48. Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103. ACM, 2008. find
  49. Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pp. 90–94. Association for Computational Linguistics, 2012. find
  50. Wiebe, Janyce, Wilson, Theresa, and Cardie, Claire. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165– 210, 2005. find
  51. Wieting, John, Bansal, Mohit, Gimpel, Kevin, and Livescu, Karen. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198, 2015. find our card
  52. Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V, Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  53. Yergeau, Francois. Utf-8, a transformation format of iso 10646. 2003. find
  54. Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014. find our card
  55. Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649–657, 2015. find our card
  56. Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Self-adaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070, 2015. — [arXiv:1504.05070](https://arxiv.org/abs/1504.05070) find
  57. Zhou, Bolei, Khosla, Aditya, Lapedriza, Agata, Oliva, Aude, and Torralba, Antonio. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:1412.6856, 2014. — [arXiv:1412.6856](https://arxiv.org/abs/1412.6856) find