Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models
What it is about
The paper turns modern English into Shakespearean English with a sentence-level encoder-decoder that has a pointer (copy) component, so it can copy words from the input. The parallel data are line-by-line modern paraphrases of sixteen Shakespeare plays, with about 18,000 training sentences and one play held out as test. Because the data are small, the authors pre-train word embeddings in two ways: with extra English text, and with a dictionary of Shakespearean-to-modern word pairs, using a retrofitting step that pulls linked words toward each other while the rest stay fixed. The best model reaches a BLEU score above 31, about six points over the strongest statistical baseline. Copying matters because many words are shared between the two styles, and the dictionary covers the archaic forms that the corpus rarely shows.
Ideas
- I1. Use the dictionary as a prior on lexicon merges. A pair of forms linked by a dictionary gets a bonus for merging into one concept in
meta.rs, and a merge is kept only if reader and encoder bits go down (open problem 2). Run it twice on dev and compare the merge set. - I2. Count, per concept, how often the standard encoder must output the source word itself. Concepts with high copy rates are the ones a delta does not need to touch. This gives a copy profile per author pair and names the words that get rewritten.
- I3. Check for gaps in the dictionary before counting merges. The paper adds the second-person forms by hand because they are frequent but missing from the dictionary. The same audit for thou, thee, ye, hath and doth would show which author features the lexicon cannot yet reach.
- I4. The neural retrofitting is not for Mova; the count form is a co-occurrence table where dictionary links act as hard constraints on merges.
More ideas (a second reading)
- I5.
en/src/morph.rs: measure how many of the 1,524 original-to-modern pairs thearchaicfunction already maps to the modern lemma, and which pairs it misses. The dictionary is a crawled list with no licence in our records, so use it for this test only. - I6.
en/src/lin.rs: an archaic surface generator as an option foren lin-eval. From a modern tree, produce thou, thee and -est forms from the lemma, and score the output against the Romeo and Juliet test set by exact match. This is a test, not a model. - I7. Copy rule: in
en/src/lin.rs, copy unchanged any word that has no dictionary entry. Measure the exact-match rate with and without the rule, on the same test set.
For Mova
The code is tools/translate (meta.rs, profile.rs). The paper is a neural system, so only its data design carries over: a small dictionary of author-to-standard pairs as a constraint on merges, and a copy profile as a measure. Open problems 2 and 6 are the targets. The reported BLEU belongs to the paper's own split and is not comparable to Mova's numbers.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-4902.pdf
Summary
The paper tackles automatic style transfer from present-day English into the Early Modern English of Shakespeare's plays. Parallel data for this task is scarce, and the two sides share much vocabulary. Yet they differ in archaic words, second-person pronoun forms, shifted word senses and spelling. The authors train on line-aligned Sparknotes paraphrases of 16 plays, a collection compiled by Xu et al., and hold out whole plays for validation and testing. Their model is an attentional sequence-to-sequence network with a bidirectional LSTM encoder. Its decoder blends a regular vocabulary softmax with a pointer network that copies input tokens, and both parts reuse the same attention weights. To offset the small corpus, word embeddings are pretrained on the training text, optionally with PTB added. They are then retrofitted toward a Shakespearean-to-modern word dictionary. The best copy-enabled configuration exceeds 31 BLEU, about 6 points above the strongest baseline. This suggests that copying plus lexical constraints suits low-resource tasks with large source–target overlap. The code is publicly released.
Key points
- Data split by play: 14 plays (18,395 sentences) for training, Twelfth Night (1,218) for validation, and Romeo and Juliet (1,462) for testing.
- On the training split, the Original side has 217K tokens and 12.39K types, while the Modern side has 200K tokens and 10.05K types. The two sides share 6.33K word types.
- The dictionary has 1,524 word pairs taken from shakespeare-words.com, with thou/thy/thyself-type forms added by hand. It is injected into the embeddings using the retrofitting method of Faruqui et al.
- Embedding variants are None, Plain, PlainExt, Retro and RetroExt, each in fixed and trainable versions. Fixed embeddings worked better, and the best setting shares embeddings between encoder and decoder.
- Evaluation uses BLEU and PINC. Baselines include statistical MT built with GIZA++ and MOSES. Neural outputs diverged more from the input (higher PINC), and an extra sentinel loss lowered BLEU by one to two points.
Interesting ideas
- Generate-or-copy mixture with shared attention. A learned gate weighs a vocabulary distribution against a copy distribution over input tokens. This fits any rewriting task where most words pass through unchanged, such as normalizing historical or dialectal text.
- Retrofitting embeddings with a bilingual word list. When parallel sentences are few, a modest dictionary of old-to-new word pairs can pull related vectors together before training. Such lists are often much cheaper to obtain than aligned corpora.
- One joint vocabulary, shared and frozen embeddings. Merging both sides' vocabularies and reusing a single embedding matrix exploits the overlap between closely related varieties. Freezing the pretrained vectors helped here, which is worth testing whenever training data is small.
- Two-phase pretraining with external text. Embeddings are trained first on external plus in-domain text, then for an equal number of iterations on in-domain text only. This is a simple way to gain coverage without losing adaptation to the target domain.
- Pairing BLEU with PINC, and splitting by work. PINC shows whether a system actually rewrites rather than echoing its input, which BLEU alone can hide when input and output overlap heavily. Holding out entire plays prevents leakage between near-duplicate contexts.
Cite
@inproceedings{jhamtani2017shakespearizing,
title = {{Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models}},
author = {Jhamtani, Harsh and Gangal, Varun and Hovy, Eduard and Nyberg, Eric},
booktitle = {Proceedings of the Workshop on Stylistic Variation},
year = {2017},
doi = {10.18653/v1/w17-4902},
url = {https://doi.org/10.18653/v1/w17-4902}
}References
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. ↗ our card
- Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. find
- Roger Brown, Albert Gilman, et al. 1960. The pronouns of power and solidarity. Article. find
- David L Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics. find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
- Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2014. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166. ↗
- Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation. In EMNLP, volume 14, pages 1532–1543. find
- Varun Gangal, Harsh Jhamtani, Graham Neubig, Eduard Hovy, and Eric Nyberg. 2017. Charmanteau: Character embedding models for portmanteau creation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark. find
- Paul Rayson, Dawn Archer, Alistair Baron, Jonathan Culpeper, and Nicholas Smith. 2007. Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora. Article. find
- Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. ↗
- Gaurush Hiranandani, Pranav Maneriker, and Harsh Jhamtani. 2017. Generating appealing brand names. arXiv preprint arXiv:1706.09335. ↗
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
- Rishiraj Saha Roy, Aishwarya Padmakumar, Guna Prasaad Jeganathan, and Ponnurangam Kumaraguru. 2015. Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text on Social Media. In 16th International Conference on Intelligent Text Processing and Computational Linguistics 2015 (CICLing ’15), pages 203–224. Springer International Publishing. find
- Eliza Kitis. 1997. Adspart of our lives: linguistic awareness of powerful advertising. Word & Image, 13(3):304–313. find
- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics. find
- H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9):e73791. find
- Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. arXiv preprint arXiv:1704.04368. ↗
- Pranava Swaroop Madhyastha, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Mapping unseen words to task-trained embedding spaces. arXiv preprint arXiv:1510.02387. ↗
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of NAACL-HLT, pages 35–40. find our card
- Balaji Vasan Srinivasan, Rishiraj Saha Roy, Harsh Jhamtani, Natwar Modani, and Niyati Chhaya. 2017. Corpus-based automatic text expansion. In CICLING. find
- Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Neural information processing systems, pages 3104–3112. find
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. find
- Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905. ↗ our card
- Wei Xu. 2014. Data-driven approaches for paraphrasing across language variations. Ph.D. thesis, New York University. find
- Wei Xu, Alan Ritter, William B Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In 24th International Conference on Computational Linguistics, COLING 2012. find our card
- Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016. Efficient Summarization with Read-Again and Copy Mechanism. arXiv preprint arXiv:1611.03382. ↗
- Barret Zoph and Kevin Knight. 2016. source neural translation. arXiv arXiv:1601.00710. Multi-preprint ↗