Style Transfer from Non-Parallel Text by Cross-Alignment

What it is about

The paper transfers style between two corpora that share one content distribution but no sentence pairs. An encoder maps a sentence, together with its style label, to a style-free latent content vector; a style-conditioned decoder renders it back. Training aligns populations rather than pairs: sentences transferred into the other style should, as a set, match the real sentences of that style, enforced by two discriminators. The method is tested on sentiment modification, on word-substitution decipherment (BLEU 57.4 at full substitution) and on word-order recovery (BLEU 26.1, about 21 points above the same model without cross-alignment). On sentiment it keeps content about as well as a control-generation baseline in human judgement (41.5% against 41.0%), while the baseline is better at flipping the label.

Ideas

I1. Population check for author deltas. For two authors, take the concept and tag profile of the delta-transferred verses and compare it, as a population, with the target author's real verses (for example a divergence over concept counts). A delta is kept only if this population distance falls, in addition to the existing rule of at least three fixed tokens. Where: the delta step in tools/translate and the check in measure.rs.
I2. Word-order recovery as a test of the linearizer. Shuffle the words of a verse (a bag of its tokens with known tree), then measure how often en/src/lin.rs restores the original order. Start with the Markov-chain baseline; the paper shows that a plain bag is hard even for a parallel model.
I3. Frequency-rank control. The decipherment baseline maps words by frequency rank. For the meta-lexicon learned on two Bible translations, a rank-matching control must be beaten before the LCS anchors and IBM steps get credit.

For Mova

The core question, how to separate content from style without pairs, is the translator's open problem 6: the standard leans to the Bibles. The population check in I1 gives a count-based way to test whether an author delta moves the output toward the target author. The paper's neural encoder is not for Mova; only the alignment criterion and the controls carry over.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.09655

Summary

The paper addresses changing the style of text, such as flipping a review's sentiment, when no paired example sentences exist. The only data are two corpora that are assumed to share a content distribution but differ in style. The authors model each sentence as generated from a latent content variable and a latent style variable. Transfer then means encoding a sentence into a style-free content vector and decoding it with the target style.

They show theoretically that this is solvable only if different styles yield distinguishable distributions, and that a richer content distribution (e.g., a Gaussian mixture) makes the mapping identifiable. Their cross-aligned auto-encoder keeps the latent content unperturbed rather than VAE-style. It also forces transferred sentences to match the real target corpus at the population level, using adversarial training with Professor-Forcing and a continuous relaxation of sampling.

The method was tested on sentiment modification, word substitution decipherment and word order recovery. It clearly outperforms baselines that lack cross-alignment. This matters because it shows distribution-level constraints alone can support controlled generation without parallel data.

Key points

On sentiment transfer of review sentences, human judges rated 41.5% of outputs as successful (sentiment changed, content kept), versus 41.0% for Hu et al.'s (2017) controllable generation approach. That baseline had higher sentiment accuracy and fluency, but this model was preferred on overall transfer quality.
On decipherment with 100% word substitution, the cross-aligned auto-encoder reaches Bleu 57.4. The aligned auto-encoder gets 7.2, the variational auto-encoder 0.9, and a parallel translation upper bound 97.2.
On word order recovery, it scores Bleu 26.1, versus 5.2 for the aligned auto-encoder and 64.6 for a model trained on parallel data. The other non-parallel models do no better than leaving the input unchanged.
The theoretical results include Proposition 1 (styles must induce different distributions) and Theorem 1 (transfer between affine transformations of a Gaussian mixture is recoverable). The authors also note that word-substitution recovery is at least as hard as graph isomorphism in general.
Code and data are released at github.com/shentianxiao/language-style-transfer (NIPS 2017).

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1705.09655

@misc{shen2017style,
  title = {{Style Transfer from Non-Parallel Text by Cross-Alignment}},
  author = {Shen, Tianxiao and Lei, Tao and Barzilay, Regina and Jaakkola, Tommi},
  year = {2017},
  eprint = {1705.09655},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.09655}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Peter F Brown, John Cocke, Stephen A Della Pietra, Vincent J Della Pietra, Fredrick Jelinek, John D Lafferty, Robert L Mercer, and Paul S Roossin. A statistical approach to machine translation. Computational linguistics, 16(2):79–85, 1990. find
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihood augmented discrete generative adversarial networks. arXiv preprint arXiv:1702.07983, 2017. ↗
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, 2016. find
Qing Dou and Kevin Knight. Large scale decipherment for out-of-domain machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 266–275. Association for Computational Linguistics, 2012. find
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016. find
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014. find
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. Differentiable scheduled sampling for credit assignment. arXiv preprint arXiv:1704.06970, 2017. ↗
R Devon Hjelm, Athul Paul Jacob, Tong Che, Kyunghyun Cho, and Yoshua Bengio. Boundary-seeking generative adversarial networks. arXiv preprint arXiv:1702.08431, 2017. ↗
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Controllable text generation. arXiv preprint arXiv:1703.00955, 2017. ↗ our card
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004, 2016. ↗
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. ↗ our card
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv:1703.05192, 2017. ↗
Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. ↗ our card
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013. ↗ our card
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810, 2017. ↗ our card
Matt J Kusner and José Miguel Hernández-Lobato. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051, 2016. ↗
Auto-encoding variational bayes. arXiv preprint Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pages 4601–4609, 2016. find
Ming-Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems, pages 469–477, 2016. find
Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017. ↗
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016. ↗ our card
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644, 2015. ↗
Jonas Mueller, Tommi Jaakkola, and David Gifford. Sequence to better sequence: continuous revision of combinatorial structures. International Conference on Machine Learning (ICML), 2017. find
Malte Nuhn and Hermann Ney. Decipherment complexity in 1: 1 substitution ciphers. In ACL (1), pages 615–621, 2013. find
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002. find our card
Allen Schmaltz, Alexander M. Rush, and Stuart Shieber. Word ordering without syntax. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2319–2324. Association for Computational Linguistics, 2016. find
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611.02200, 2016. ↗
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
Zili Yi, Hao Zhang, Ping Tan Gong, et al. Dualgan: Unsupervised dual learning for image-to-image translation. arXiv preprint arXiv:1704.02510, 2017. ↗
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: sequence generative adversarial nets with policy gradient. arXiv preprint arXiv:1609.05473, 2016. ↗
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593, 2017. ↗