Elamur Research⌕
☰
home / papers / 2026 / Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning

Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning

Paper · 2026 · Timon Ziegenbein, Maja Stahl and Henning Wachsmuth

Summary

LLMs used to make arguments more appropriate tend to edit differently from people. They scatter many small, interdependent changes across a text and often shift its meaning. Human editors instead group related changes into contained, meaning-preserving edits that can be accepted or rejected one at a time. The authors use reinforcement learning to teach an LLM this human style. The policy, Llama-3.1-8B-Instruct trained with group relative policy optimization (GRPO), proposes sentence-level edit suggestions for an argument. Its reward balances two goals. The first is edit-level human-likeness, checked by three binary classifiers for semantic similarity, fluency and conformity to typical keep/delete/add/substitute patterns. The second is improvement in argument-level appropriateness. The experiments draw on the corpus of Ziegenbein et al. (2023), in which each argument carries labels marking whether and how it is inappropriate. In automatic and human evaluation, the method beats strong baselines, including the earlier RL full-rewriting system of Ziegenbein et al. (2024a). With repeated editing rounds, its appropriateness gains come close to those of full rewriting. This matters for writing-assistance tools, where users need suggestions they can apply selectively.

Key points

  • Human-likeness criteria: an edit counts as human-like only if all three sentence-level classifiers approve it:

    • semantic similarity to the original sentence,
    • no loss of fluency,
    • edit operations that match typical human patterns.

    The pattern-conformity criterion is new, and the paper presents it as the first edit-level version of style-transfer metrics.

  • Setup: GRPO training of Llama-3.1-8B-Instruct, with no parallel human-edit data required.

  • Data: the appropriateness-annotated argument corpus and taxonomy of Ziegenbein et al. (2023). Inappropriateness covers toxic emotions, lack of commitment and lack of intelligibility. All experiments are in English.

  • Results: human evaluators prefer the full model (GRPO full) at both the edit and the argument level. Ablations show that each of the three edit-level components contributes to human-like editing.

  • Resources: code and data are released at github.com/timonziegenbein/inappropriateness-editing. The authors recommend human-in-the-loop use because appropriateness is subjective and culture-dependent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Leibniz University HannoverHanover · Germany6 works · 599 citations🔬L3S Research CenterHanover · Germany4 works · 4 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Timon Ziegenbein, Maja Stahl and Henning Wachsmuth. 2026. Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1789

@inproceedings{ziegenbein2026teaching,
  title = {{Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning}},
  author = {Timon Ziegenbein, Maja Stahl and Henning Wachsmuth},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1789},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1789}
}

References

58 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn Walker. 2016. Internet argument corpus 2.0: An SQL schema for dialogic social media and the corpora to go with it. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4445–4452, Portorož, Sloven… find
  2. Rohan Anil, Sebastian Borgeaud, et al. 2025. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.
  3. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  4. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4302–4310, Red Hook, NY, USA. Curran Associa… find
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodalit…
  6. Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393–3402, Florence, Ita…
  7. Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022. Understanding iterative revision from human-written text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573–3590, Dublin, Ireland.… find
  8. Linda S Flower and John R Hayes. 2016. The dynamics of composing: Making plans and juggling constraints. In Cognitive processes in writing, pages 31–50. Routledge. find
  9. Allan Gollins and Dedre Gentner. 2016. A framework for a cognitive theory of writing. In Cognitive processes in writing, pages 51–72. Routledge. find
  10. Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  12. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  13. Xinlei He, Savvas Zannettou, Yun Shen, and Yang Zhang. 2023. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content. Preprint, arXiv:2308.05596. — [arXiv:2308.05596](https://arxiv.org/abs/2308.05596) find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  15. Thomas Huber and Christina Niklaus. 2025. CLEAR: A comprehensive linguistic evaluation of argument rewriting by large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 19548–19568, Suzhou, China. Association for Computational Linguistics. find
  16. Chao Jiang, Wei Xu, and Samuel Stevens. 2022. arXivEdits: Understanding the human revision process in scientific writing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, Abu Dhabi, United Arab Emirates. Association for Computational Linguis… find
  17. Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Asso…
  18. Léo Laugier, John Pavlopoulos, Jeffrey Sorensen, and Lucas Dixon. 2021. Civil rephrases of toxic texts with self-supervised transformers. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1442–1461, Online. Association… find
  19. Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
  20. Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5116–5122. Internatio…
  21. Karthic Madanagopal and James Caverlee. 2023. Reinforced sequence training based subjective bias correction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2585–2598, Dubrovnik, Croatia. Association for Computational Linguistics. find
  22. Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. FELIX: Flexible text editing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244–1255, Online. Association for Computational Linguistics. find
  23. Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: High-precision text editing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces…
  24. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pa…
  25. Lily Ng, Anne Lauscher, Joel Tetreault, and Courtney Napoles. 2020. Creating a domain-diverse corpus for theory-based argument quality assessment. In Proceedings of the 7th Workshop on Argument Mining, pages 117–126, Online. Association for Computational Linguistics. find
  26. Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 189–194, Melbourne, Australia.…
  27. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. A… find
  28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  29. Vipul Raheja, Dimitris Alikaniotis, Vivek Kulkarni, Bashar Alhafni, and Dhruv Kumar. 2024. mEdIT: Multilingual text editing via instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  30. Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text editing by task-specific instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5274–5291, Singapore. Association for Computational Linguistics. find
  31. Joni O. Salminen, Hind Almerekhi, Milica Milenkovic, Soon-Gyo Jung, Jisun An, Haewoon Kwak, and B. Jansen. 2018. Anatomy of online hate: Developing a taxonomy and machine learning models for identifying and classifying hate in online news media. International Conference on Web and Social Media. find
  32. Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus Mar… find
  33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  35. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, and Lei Meng. 2024. RewriteLM: an instruction-tuned large language model for text rewriting. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innova… find
  36. Gabriella Skitalinskaya, Maximilian Spliethöver, and Henning Wachsmuth. 2023. Claim optimization in computational argumentation. In Proceedings of the 16th International Natural Language Generation Conference, pages 134–152, Prague, Czechia. Association for Computational Linguistics. find
  37. Maja Stahl, Timon Ziegenbein, Joonsuk Park, and Henning Wachsmuth. 2025. ArgInstruct: Specialized instruction fine-tuning for computational argumentation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11103–11127, Vienna, Austria. Association for Computational Linguis… find
  38. Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit operations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5147–5159, Online. Association for Computational Linguistics.
  39. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. find
  40. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  41. Marie M. Vaughan and David D. McDonald. 1986. A model of revision in natural language generation. In Proceedings of the 24th Annual Meeting on Association for Computational Linguistics, ACL ’86, page 90–96, USA. Association for Computational Linguistics. find
  42. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberd-ingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu…
  43. Henning Wachsmuth and Till Werner. 2020. Intrinsic quality assessment of arguments. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6739–6745, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  44. Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 812– 817, Istanbul, Turkey. European Language Resources Associ…
  45. D. Walton. 2010. The Place of Emotion in Argument. Pennsylvania State University Press. find
  46. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  47. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e…
  48. Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4873–4883, Florence, Italy. Association for Computational Li… find
  49. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, pages 1391–1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. find
  50. Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  51. Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in Wikipedia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000–2010, Copenhagen, Denmark. Association for Computational Linguistics… find
  52. Yiming Zeng, Wanhao Yu, Zexin Li, Tao Ren, Yu Ma, Jinghan Cao, Xiyan Chen, and Tingting Yu. 2025. Fineedit: Unlock instruction-based text editing for llms. arXiv preprint arXiv:2502.13358. find
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  54. Yiqun Zhang, Mingjie Zhao, Yunfan Zhang, and Yiu-ming Cheung. 2025. Trending applications of large language models: A user perspective survey. IEEE Transactions on Artificial Intelligence, 1(01):1–17. find
  55. Timon Ziegenbein, Gabriella Skitalinskaya, Alireza Bayat Makou, and Henning Wachsmuth. 2024a. LLM-based rewriting of inappropriate argumentation using reinforcement learning from machine feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L… find
  56. Timon Ziegenbein, Shahbaz Syed, Felix Lange, Martin Potthast, and Henning Wachsmuth. 2023. Modeling appropriate language in argumentation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4344–4363, Toronto, Canada. Association… find
  57. Timon Ziegenbein, Shahbaz Syed, Martin Potthast, and Henning Wachsmuth. 2024b. Objective argument summarization in search. In Robust Argumentation Machines, pages 335–351, Cham. Springer Nature Switzerland. find
  58. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ziegenbein-2026-teaching-llms-human-like-editing/