Elamur Research⌕
☰
home / papers / 2026 / SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task

SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task

Paper · 2026 · Claudio Savelli, Moreno La Quatra, Alkis Koudounas et al.

Summary

SVELA, a shared task at EVALITA 2026, addresses how to audit machine unlearning. The question is whether a Large Language Model that has been through an unlearning procedure has actually lost targeted knowledge, which matters for GDPR-style deletion requests. The organizers fine-tuned LLMs on synthetic actor biographies from the FAME dataset, which covers Italian, Spanish, French and German. They then applied undisclosed unlearning methods.

Participants had to build black-box or white-box verifiers that label information as retain, forget or never-used. This was done at two levels:

  • Subtask 1: whole identities.
  • Subtask 2: single question-answer facts.

Systems were submitted as runnable code and executed on both public and hidden models, so generalization was tested fairly. Scoring used macro-F1.

Key findings:

  • Fact-level verification was clearly harder than identity-level verification.
  • Retained knowledge was easy to spot.
  • Forgotten and never-seen content were hard to separate, because unlearning makes a model respond as if it never saw the data.

Overall performance was modest, so reliable unlearning verification remains an open problem.

Key points

  • Participation: eight teams registered, four wrote system papers, and there were more than fifty valid submissions across both subtasks.
  • Data: each FAME identity has 20 atomic facts, each mapped to one QA pair, in four groups: Biography (5), Career (7), Achievements (3) and Personal (5).
  • Models and methods: verifiers were evaluated on 1B and 3B models, with 3B giving small but steady macro-F1 gains. The unlearning methods included GA, GD, KLM and PO.
  • Top result: the team priyam_saha17 ranked first in both subtasks for both model sizes.
  • Method comparison: PO (preference-based unlearning) gave slightly more detectable forgetting, likely because it teaches the model to abstain with answers like "I don't know."

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Politecnico di TorinoTurin · Italy11 works · 18 citations🎓Università degli Studi di Enna KoreEnna · Italy2 works · 6 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Claudio Savelli, Moreno La Quatra, Alkis Koudounas and Flavio Giobergia. 2026. SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{savelli2026svela,
  title = {{SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task}},
  author = {Claudio Savelli, Moreno {La Quatra}, Alkis Koudounas and Flavio Giobergia},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.42/}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, C. Zhang, Quantifying memorization across neural language models, in: The Eleventh International Conference on Learning Representations, 2022. find
  2. A. Mantelero, The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’, Computer Law & Security Review 29 (2013) 229–235. find
  3. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, in: Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 30016–300…
  4. M. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, N. Papernot, Machine unlearning, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2021, pp. 141–159. doi: 10.1109/SP40001.2021.00018 . — [doi:10.1109/sp40001.2021.00018](https://doi.org/10.1109/sp40001.2021.00018) find
  5. S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, et al., Rethinking machine unlearning for large language models, Nature Machine Intelligence (2025) 1–14. find
  6. J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, X. Yue, Machine unlearning of pre-trained large language models, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computat…
  7. A. Koudounas, C. Savelli, F. Giobergia, E. Baralis, “Alexa, can you forget me?” Machine Unlearning Benchmark in Spoken Language Understanding, in: Interspeech 2025, 2025, pp. 1768–1772. doi: 10.21437/Interspeech.2025- 2607 . — [doi:10.21437/interspeech.2025-](https://doi.org/10.21437/interspeech.2025-) find
  8. D. Andrea, S. Claudio, T. Gabriele, G. Flavio, B. Elena, G. Stilo, et al., How to make reproducible research in machine unlearning with erasure, in: Proceedings of the Thirty-Fourth International Joint Conference onArtificial Intelligence,{IJCAI-25}, 2025, pp. 11025–11029. find
  9. A. D’Angelo, C. Savelli, G. Tagliente, F. Giobergia, E. Baralis, G. Stilo, Erasure: A modular and extensible framework for machine unlearning, in: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 6346–6350. find
  10. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… find
  11. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, FAME: Fictional actors for multilingual erasure, in: Proceedings of the Fifteenth Language Resources and Evaluation Conference, European Language Resources Association, 2026.
  12. A. Golatkar, A. Achille, S. Soatto, Eternal sunshine of the spotless net: Selective forgetting in deep networks, in: CVPR, 2020. find
  13. D. Choi, D. Na, Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems, arXiv preprint arXiv:2311.02240 (2023). — [arXiv:2311.02240](https://arxiv.org/abs/2311.02240) find
  14. M. Kurmanji, P. Triantafillou, J. Hayes, E. Triantafillou, Towards unbounded machine unlearning, NeurIPS 36 (2024). find
  15. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, J. Z. Kolter, Tofu: A task of fictitious unlearning for llms, arXiv preprint arXiv:2401.06121 (2024).
  16. H. Xu, T. Zhu, L. Zhang, W. Zhou, P. S. Yu, Machine unlearning: A survey, ACM Comput. Surv. 56 (2023). URL: https://doi.org/10.1145/3603620. doi: 10.1145/3603620 . — [doi:10.1145/3603620](https://doi.org/10.1145/3603620) ↗
  17. P. Saha, priyam_saha17 at svela: A feature-centric pipeline for verifying selective forgetting in large language models, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026.
  18. E. Munis, M. Sabato, E. Bayat, A. Lolli, Malto at svela: A specific-attention-head approach for membership inference attacks in llms unlearning evaluation, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEU…
  19. A. Yassine, H. Ibrahim, L. Cagliero, Ita-lib at svela: Detecting the forgotten — representation-based approach for verifying machine unlearning, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari…
  20. M. Berta, T. Cerquitelli, Eraserhead at svela: Detecting llm forgetting via logit-space statistics, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026. find
  21. W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, C. Zhang, Muse: Machine unlearning six-way evaluation for language models, arXiv preprint arXiv:2407.06460 (2024). find
  22. C. Savelli, E. Munis, E. Bayat, A. Grieco, F. Giobergia, Malto at semeval-2025 task 4: Dual teachers for unlearning sensitive content in llms, in: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), 2025, pp. 1747–1752. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/