Elamur Research
☰
home / papers / 2012 / Paraphrasing for Style

Paraphrasing for Style

What it is about

The paper asks whether a paraphrase system can be steered toward a writing style. Its testbed is Shakespeare's plays paired with their modern prose translations, aligned into roughly 21,000 sentence pairs from one source and about 10,000 from another. A phrase-based statistical MT system is trained in the modern-to-Shakespearean direction and compared with a dictionary-substitution baseline and with paraphrase tables built from out-of-domain text. The authors also ask how well the automatic metrics measure style: BLEU and PINC are joined by three new style scores (cosine similarity, a language model, and logistic regression) that are checked against human judgments. The parallel-trained systems beat both baselines. BLEU tends to mix meaning with style, while the new style scores agree better with people.

Ideas

For Mova

The code is tools/translate (profile.rs, measure.rs). The paper's phrase-based SMT is not for Mova, but its two-baseline design (substitution and out-of-domain text) is the right control for open problem 6, where the standard leans toward the Bibles. The style scores are the measure to add in normal mode, since exact match alone says nothing about style.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/C12-1177.pdf

Summary

This COLING 2012 paper presents what the authors describe as the first study of paraphrasing aimed at a target writing style rather than at meaning alone. The testbed is Shakespeare's Early Modern English, chosen because modern "translations" of his plays exist. The authors built parallel corpora by scraping modern versions of the plays from Sparknotes and Enotes and sentence-aligning them with the originals. On this data they trained a phrase-based MT system (GIZA++ and Moses). They compared it with two baselines that need no in-style parallel text: one uses a Shakespearean dictionary as a phrase table, and the other uses an out-of-domain video-caption paraphrase corpus. Both baselines are combined with a language model built from Shakespeare's plays. Evaluation used BLEU, PINC and human ratings of meaning, dissimilarity and style. The system trained on the modest parallel data beat both baselines. The authors also found that BLEU mixes up meaning and style, so they proposed three style-specific metrics that agree better with human judgments. The work matters because it frames style transfer as a measurable paraphrase task.

Key points

  • Sparknotes corpus: modern versions of 17 plays. Alignment kept 21,079 of 31,718 sentence pairs.
  • Enotes corpus: modern versions of 8 of those plays. Alignment kept 10,365 of 13,640 pairs.
  • Copying the input unchanged (BLEU against the original): Enotes scores 52.30 and Sparknotes 24.67. Enotes is therefore much more conservative, so evaluation focuses on Sparknotes.
  • Held-out data: Romeo and Juliet was excluded from all training. Human evaluation used 100 randomly sampled lines from it.
  • Dictionary baseline: 68,709 word/phrase pairs scraped from shakespeareswords.com, plus a language model built with SRILM on 36 plays.
  • Video-corpus baseline: this system changed very little of the input and got a low PINC score. Many archaic words had no match in its phrase table.
  • MERT tuning: it made the parallel-data systems more cautious (lower PINC) without raising test BLEU.
  • New style metrics: three metrics based on cosine similarity, language models and logistic regression. They correlated better with human judgments than BLEU and PINC.

Interesting ideas

  • Reuse study aids as parallel data. Educational "translations" written for students can serve as ready-made monolingual parallel corpora. The same trick could apply to other registers, such as simplified news or plain-language legal summaries.
  • Turn a dictionary into a phrase table. When a word list has no probabilities, translation weights can be estimated by scoring each target-side option with a target-style language model and normalizing per source entry. This yields a usable MT translation model from a bare lexicon.
  • Pair a generic phrase table with an in-style language model. Out-of-domain paraphrase pairs can still be pushed toward a target style this way. The paper also shows the main weakness: coverage fails when the source vocabulary is archaic or unusual.
  • Report a "no change" baseline. Scoring the input copied unchanged against the references shows how conservative a reference set is. This helps choose which references to evaluate against.
  • Evaluate style separately from meaning. Asking human raters about semantic adequacy, lexical dissimilarity and style as separate questions shows where BLEU hides trade-offs. Dedicated style metrics can then be validated against those separate ratings in any style-transfer task.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{xu2012paraphrasing,
  title = {{Paraphrasing for Style}},
  author = {Xu, Wei and Ritter, Alan and Dolan, Bill and Grishman, Ralph and Cherry, Colin},
  booktitle = {Proceedings of COLING 2012},
  year = {2012},
  url = {https://aclanthology.org/C12-1177/}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bannard, C. and Callison-Burch, C. (2005). Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). find our card
  2. Barzilay, R. and Lee, L. (2003). Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03. find our card
  3. Callison-Burch, C. (2008). Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  4. Callison-Burch, C., Cohn, T., and Lapata, M. (2008). Parametric: an automatic evaluation metric for paraphrasing. In Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1. find
  5. Chandrasekar, R., Doran, C., and Srinivas, B. (1996). Motivations and methods for text simplification. In Proceedings of the 16th conference on Computational linguistics - Volume 2. find
  6. Chen, D. L. and Dolan, W. B. (2011). Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011), Portland, OR. find
  7. Clarke, J. and Lapata, M. (2008). Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, 31(1):399–429. find
  8. Cohn, T. and Lapata, M. (2009). Sentence compression as tree transduction. Journal of Artificial Intelligence Research, 34:637–674. find
  9. Das, D. and Smith, N. A. (2009). Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1, ACL ’09. find
  10. Dolan, B., Quirk, C., and Brockett, C. (2004). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of Coling 2004. find our card
  11. Gamon, M. (2004). Linguistic correlates of style: authorship classification with deep linguistic analysis features. In Proceedings of the 20th international conference on Computational Linguistics. find
  12. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. (2007). The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07. find
  13. Greene, E., Bodrumlu, T., and Knight, K. (2010). Automatic analysis of rhythmic poetry with applications to generation and translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. find
  14. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions. find
  15. Koehn, P. and Knight, K. (2000). Estimating word translation probabilities from unrelated monolingual corpora using the em algorithm. In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence. find
  16. Kok, S. and Brockett, C. (2010). Hitting the right paraphrases in good time. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  17. Liu, C., Dahlmeier, D., and Ng, H. T. (2010). PEM: A paraphrase evaluation metric exploiting parallel texts. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. find
  18. Madnani, N. and Dorr, B. (2010). Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics, 36(3):341–387. find our card
  19. Moore, R. C. (2002). Fast and accurate sentence alignment of bilingual corpora. In Proceedings of the 5th Conference of the Association for Machine Translation in the Americas on Machine Translation: From Research to Real Users, AMTA ’02. find
  20. Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1. find
  21. Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Comput. Linguist. find
  22. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. find our card
  23. Quirk, C., Brockett, C., and Dolan, W. (2004). Monolingual machine translation for paraphrase generation. In Proceedings of EMNLP 2004. find our card
  24. Raghavan, S., Kovashka, A., and Mooney, R. (2010). Authorship attribution using probabilistic context-free grammars. In Proceedings of the ACL 2010 Conference Short Papers. find
  25. Shinyama, Y. and Sekine, S. (2003). Paraphrase acquisition for information extraction. In Proceedings of the second international workshop on Paraphrasing - Volume 16, PARAPHRASE ’03. find
  26. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08. find
  27. Stolcke, A. (2002). Srilm - an extensible language modeling toolkit. In INTERSPEECH. find
  28. Vanderwende, L., Suzuki, H., Brockett, C., and Nenkova, A. (2007). Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. Inf. Process. Manage. find
  29. Yatskar, M., Pang, B., Danescu-Niculescu-Mizil, C., and Lee, L. (2010). For the sake of simplicity: unsupervised extraction of lexical simplifications from wikipedia. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find