Elamur Research⌕
☰
home / papers / 2026 / ReproHum #0669-08: Reproducing a Recipe for Arbitrary Text Style Transfer with LLMs

ReproHum #0669-08: Reproducing a Recipe for Arbitrary Text Style Transfer with LLMs

Summary

This ReproNLP-style study re-runs one human evaluation from Reif et al. (2022), which proposed augmented zero-shot prompting for arbitrary text style transfer with LLMs. The reproduction targets a single criterion: semantic preservation for the "more positive" style. It covers four output types: human-written transfers, zero-shot, augmented zero-shot and paraphrase prompts. The original authors supplied the data, but their internal annotation interface was unavailable, so the author rebuilt it with GPT-4 from a screenshot in the paper. Six Prolific crowdworkers replaced the original six professional annotators.

Scores for the paraphrase and zero-shot prompts fell sharply compared with the original. Augmented zero-shot and human outputs were more stable. However, augmented zero-shot did not match the human baseline, which echoes the parallel reproduction by Onderková et al. (2025). The author attributes the gap to the different annotator cohorts and the recreated interface. The paper argues that original studies should keep and share supporting materials such as evaluation interfaces.

Key points

  • Setup: 50 Reddit Writing Prompts sentences were used, giving 300 items in total. Each item received 3 ratings and each participant rated 150 items. Ratings used a 1–100 scale, and pay was £12.60/hour (UK living wage). A HEDS datasheet was completed.
  • Mean scores, original vs. reproduction: Original values were estimated from bar plots via pixel counting.
    • paraphrase: 90.29 → 21.74
    • zero-shot: 69.71 → 24.24
    • augmented zero-shot: 86.47 → 67.31
    • human: 85.29 → 79.33
  • Two-study CV* values: The human outputs were the most reproducible at 6.5. The other values were 110.43 for paraphrase, 87.65 for zero-shot and 22.37 for augmented zero-shot.
  • Three-study comparison: The QRA tool (Belz and Thomson, 2026) was applied across the original study, Onderková et al. (2025) and this reproduction. It showed the same pattern, with closer agreement for augmented zero-shot and human outputs.
  • Main finding: The original claim that augmented zero-shot is comparable to human transfers could not be confirmed in either reproduction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.90

@inproceedings{mahamood2026reprohum,
  title = {{ReproHum \#0669-08: Reproducing a Recipe for Arbitrary Text Style Transfer with LLMs}},
  author = {Mahamood, Saad},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.90},
  url = {https://doi.org/10.18653/v1/2026.gem-main.90}
}

References

24 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In 11th conference of the European chapter of the association for computational linguistics, pages 313–320. find
  3. Anja Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021a. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Generation, pages 249–258. find
  4. Anja Belz and Craig Thomson. 2023. The 2023 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results. In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, pages 35–48. find
  5. Anya Belz. 2022. A Metrological Perspective on Reproducibility in NLP*. Computational Linguistics, 48(4):1125–1135. ↗
  6. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021b. A systematic review of reproducibility research in natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 381–393, Online. Association for Computational Linguistics. ↗
  7. Anya Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021c. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Generation, pages 249–258, Aberdeen, Scotland, UK. Association for Computational Linguistics. find
  8. Anya Belz, Anastasia Shimorina, Maja Popović, and Ehud Reiter. 2022. The 2022 ReproGen shared task on reproducibility of evaluations in NLG: Overview and results. In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, pages 43–51, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics. find
  9. Anya Belz and Craig Thomson. 2024. The 2024 ReproNLP shared task on reproducibility of evaluations in NLP: Overview and results. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 91–105, Torino, Italia. ELRA and ICCL. find
  10. Anya Belz and Craig Thomson. 2026. Quantified reproducibility assessment for four common types of evaluation results in nlp/ml. Computational Linguistics, pages 1–10. ↗
  11. Anya Belz, Craig Thomson, and Javier González Corbelle. 2026. The shared task on reproducibility of evaluations in nlp (ReproNLP) 2026: Overview and results. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM²), San Diego, USA. Association for Computational Linguistics. find
  12. Anya Belz, Craig Thomson, Javier González Corbelle, and Malo Ruelle. 2025. The 2025 ReproNLP shared task on reproducibility of evaluations in NLP: Overview and results. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 1002–1016, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  13. Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ondřej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González-Corbelle, Dirk Hovy, and 23 others. 2023a. Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 1–10, Dubrovnik, Croatia. Association for Computational Linguistics. find
  14. Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023b. Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Computational Linguistics. find
  15. Eduardo Calò, Lydia Penkert, and Saad Mahamood. 2025. Lessons from a user experience evaluation of NLP interfaces. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2915–2929, Albuquerque, New Mexico. Association for Computational Linguistics. find
  16. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
  17. Rudali Huidrom and Anya Belz. 2025. Using LLM judgements for sanity checking results and reproducibility of human evaluations in NLP. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 354–365, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  18. Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), pages 5116–5112, Macao, China. International Joint Conferences on Artificial Intelligence. find our card
  19. Saad Mahamood. 2024. ReproHum #0124-03: Reproducing human evaluations of end-to-end approaches for referring expression generation. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 250–254, Torino, Italia. ELRA and ICCL. find
  20. Kristýna Onderková, Mateusz Lango, Patrícia Schmidtová, and Ondrej Dusek. 2025. ReproHum #0669-08: Reproducing sentiment transfer evaluation. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 601–608, Vienna, Austria and virtual meeting. Association for Computational Linguistics. find
  21. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866–876, Melbourne, Australia. Association for Computational Linguistics. find our card
  22. Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837–848, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  23. Anastasia Shimorina and Anya Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75, Dublin, Ireland. Association for Computational Linguistics. ↗
  24. Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common Flaws in Running Human Evaluation Experiments in NLP. Computational Linguistics, pages 1–11. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mahamood-2026-reprohum-0669-08-reproducing-recipe/