Elamur Research⌕
☰
home / papers / 2026 / Par-ITA: Benchmarking Seq2Seq and LLMs on a Human-Supervised Parallel Corpus for Italian Hyperpartisan Neutralization

Par-ITA: Benchmarking Seq2Seq and LLMs on a Human-Supervised Parallel Corpus for Italian Hyperpartisan Neutralization

Paper · 2026 · Michele Joshua Maggini, Søren Fomsgaard, Michele Maestroni et al.

Summary

Most work on rewriting hyperpartisan text into neutral language has targeted English, and Italian has had no dedicated resource. The authors built Par-ITA from the Italian portion of the SemEval 2023 Task 3 news data. It contains 2,475 paragraph pairs created under human supervision.

Construction followed three stages:

  1. Three open LLMs were compared, and Qwen2.5-14B-Instruct was selected as the generator.
  2. Qwen produced neutral and hyperpartisan variants via backtranslation, and experts then corrected the outputs heavily.
  3. Human experts validated the resulting models afterwards.

The authors then benchmarked seq2seq models (e.g., IT5, BART) and decoder-only LLMs using fine-tuning, DPO and in-context learning. DPO gave seq2seq systems the highest neutrality scores. Automatic metrics, however, did not fully match human judgments.

GPT-4o-mini, tested as an automatic judge, tended to label texts on sensitive political topics as hyperpartisan more often than experts did. This suggests human oversight is still needed. Beyond the dataset, the work offers a reusable recipe for building corpora in subjective, non-English tasks.

Key points

  • Two native-speaker annotators identified 825 hyperpartisan paragraphs, with an initial Cohen's κ of 0.85. The full effort took about 300 hours across three custom web interfaces.
  • In Stage 1, Mistral-Nemo-2407, Llama-3.1-8b-Instruct and Qwen2.5-14B-Instruct were compared. Qwen failed to neutralize in only 13% of cases, against 70% for Mistral. Llama often drifted into English.
  • Humans edited 32–68% of the generated outputs, and the average Levenshtein distance from the original was 25–32%.
  • Agreement between GPT-4o-mini and humans on neutrality labels was poor to fair (κ from −0.039 to 0.454). It over-flagged topics such as COVID, immigration and climate change.
  • For evaluation, the authors suggest using BLEU and neutrality together. Systems scoring above 10 BLEU and above 0.90 neutrality consistently received human ratings of at least 3.0 out of 5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Universidade de Santiago de CompostelaSantiago de Compostela · Spain13 works · 364 citations🎓Université de Caen NormandieCaen · France3 works🔬GREYCCaen · France2 works🎓École Nationale Supérieure d'Ingénieurs de CaenCaen · France2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

Cite

Cite as Michele Joshua Maggini, Søren Fomsgaard, Michele Maestroni, Gaël Dias and Pablo Gamallo. 2026. Par-ITA: Benchmarking Seq2Seq and LLMs on a Human-Supervised Parallel Corpus for Italian Hyperpartisan Neutralization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.253

@inproceedings{maggini2026par,
  title = {{Par-ITA: Benchmarking Seq2Seq and LLMs on a Human-Supervised Parallel Corpus for Italian Hyperpartisan Neutralization}},
  author = {Michele Joshua Maggini, Søren Fomsgaard, Michele Maestroni, Gaël Dias and Pablo Gamallo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.253},
  url = {https://doi.org/10.18653/v1/2026.acl-long.253}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 5600 9 others. 202…
  2. Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, and 1 others. 2025. Apertus: Democratizing open and compliant llms for global language environments. arXiv prep…
  3. Katherine Atwell, Sabit Hassan, and Malihe Alikhani. 2022. APPDIA: A discourse-aware transformer-based style transfer model for offensive social media conversations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6063–6074, Gyeongju, Republic of Korea. Intern… find
  4. Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza del Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large language model hacking: Quantifying the hidden risks of using llms for text annotation. Preprint, arXiv:2509.08825. — [arXiv:2509.08825](https://arxiv.org/abs/2509.08825) find
  5. Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. Sensitivity, performance, robustness: Deconstructing the effect of sociodemographic prompting. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),…
  6. Michael Brüggemann and Hendrik Meyer. 2023. When debates break apart: Discursive polarization as a multi-dimensional divergence emerging in and through communication. Communication Theory, 33(2-3):132–142. find
  7. Viola De Ruvo, Arianna Muti, Daryna Dementieva, and Debora Nozza. 2025. Detoxify-IT: An Italian parallel dataset for text detoxification. In Proceedings of the The 9th Workshop on Online Abuse and Harms (WOAH), pages 267–275, Vienna, Austria. Association for Computational Linguistics. find
  8. Daryna Dementieva, Nikolay Babakov, and Alexander Panchenko. 2024a. Multiparadetox: Extending text detoxification with parallel data to new languages. arXiv preprint arXiv:2404.02037. find
  9. Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Frol-ian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, and 1 others. 2024b. Overview of the multilingual text detoxification task at pan 2024. Working Notes of CLEF. find
  10. Miguel Fernández, Maximiliano Ojeda, Lilly Guevara, Diego Varela, Marcelo Mendoza, and Alberto Barrón-Cedeño. 2024. Victor vectors@ dipromats 2024: Propaganda detection with llm paraphrasing and machine translation. In Proceedings of the Iberian languages evaluation forum (IberLEF 2024) co-located w… find
  11. Michael Heseltine and Bernhard Clemm von Hohenberg. 2024. Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1):20531680241236239. find
  12. Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, and Timo Spinde. 2025. The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 137… find
  13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, and 16 others. 2024.… find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  15. David Dale, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Text detoxification using large pre-trained neural models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7979–7996, Online… find
  16. Lucas Raniére Juvino Santos, Leandro Balby Marinho, Claudio Elizio Calazans Campelo, Filippo Menczer, and Alessandro Flammini. 2025. Can large language models effectively mitigate polarization in social media text? In Proceedings of the 17th ACM Web Science Conference 2025, Websci ’25, page 348–357,… find
  17. Ernesto de León, Mykola Makhortykh, and Silke Adam. 2024. Hyperpartisan, alternative, and conspiracy media users: An anti-establishment portrait. Political Communication, 41(6):877–902. find
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training Laura Iannelli, Bianca Biagi, and Marta Meleddu. 2021. Public opinion polarization on immigration in italy: the r… find
  19. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021a. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14857– 14866. find
  20. Ruibo Liu, Lili Wang, Chenyan Jia, and Soroush Vosoughi. 2021b. Political depolarization of news articles using attribute-aware word embeddings. CoRR, abs/2101.01391. — [arXiv:2101.01391](https://arxiv.org/abs/2101.01391) find
  21. Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
  22. Michele Joshua Maggini, Davide Bassi, and Pablo Gamallo. 2025a. Detecting hyperpartisanship and rhetorical bias in climate journalism: A sentence-level Italian dataset. In Proceedings of the 2nd Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2025), pages 168–187, Vienna, Au… find
  23. Michele Joshua Maggini, Davide Bassi, Paloma Piot, Gaël Dias, and Pablo Gamallo Otero. 2025b. A systematic review of automated hyperpartisan news detection. PloS one, 20(2):e0316989. find
  24. Michele Joshua Maggini, Davide Bassi, Paloma Piot, Gaël Dias, and Pablo Gamallo Otero. 2025c. A systematic review of automated hyperpartisan news detection. PLOS ONE, 20(2):1–39. find
  25. Erik Bran Marino, Jesus M Benitez-Baleato, and Ana Sofia Ribeiro. 2024. The polarization loop: How emotions drive propagation of disinformation in online media—the case of conspiracy theories and extreme right movements in southern europe. Social Sciences, 13(11):603. find
  26. Mistral. 2024. Mistral nemo: Collaborative innovation with nvidia. find
  27. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2… find
  28. Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. find
  29. Automatically neutralizing subjective bias in text. In Proceedings of the aaai conference on artificial intelligence, volume 34, pages 480–489. find
  30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  31. Amy Ross Arguedas, Craig Robertson, Richard Fletcher, and Rasmus Nielsen. 2022. Echo chambers, filter bubbles, and polarisation: A literature review. find
  32. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  33. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap…
  34. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  35. Gabriele Sarti and Malvina Nissim. 2024. IT5: Text-to-text pretraining for Italian language understanding and generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9422–9433, Torino, Italia. ELR… find
  36. Kasidit Sermsri and Teerapong Panboonyuen. 2025. Debiasing large language models in thai political stance detection via counterfactual calibration. In Proceedings of the 9th Widening NLP Workshop, pages 56– 64. find
  37. Jennifer Stromer-Galley, Brian McKernan, Saklain Zaman, Chinmay Maganur, and Sampada Regmi. 2025. The efficacy of large language models and crowd annotation for accurate content analysis of political social media messages. Social Science Computer Review, 0(0):08944393251334977. find
  38. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  39. Martina Toshevska and Sonja Gievska. 2025. Llm-based text style transfer: Have we taken a step forward? IEEE Access, 13:44707–44721. find
  40. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, a…
  41. Petter Törnberg. 2024. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, 0(2024):08944393241286471. find
  42. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 conference of the North American chapter of the association for computational ling…
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maggini-2026-par-ita-benchmarking-seq2seq-llms/