Elamur Research⌕
☰
home / papers / 2026 / priyam_saha17 at SVELA: A Feature-Centric Pipeline for Verifying Selective Forgetting in Large Language Models

priyam_saha17 at SVELA: A Feature-Centric Pipeline for Verifying Selective Forgetting in Large Language Models

Paper · 2026 · Priyam Saha

Summary

The SVELA shared task at EVALITA 2026 tests whether machine unlearning can be audited from the outside. Organizers fine-tuned LLMs and then made them unlearn a hidden set of identities. For a given identity–topic pair, a system must decide whether the model kept the knowledge, forgot it, or never saw it.

This entry treats the audit as a three-way supervised classification problem over the behaviour of frozen models. It makes no parameter changes and no gradient updates. Each multilingual question is wrapped in a fixed English instruction prompt. The pipeline then extracts hidden-state embeddings and uncertainty statistics, including predictive entropy and logit margin. These features are compressed with IPCA and passed to a shallow residual classifier.

Across every one of the four tracks (two tasks × 1B/3B checkpoints), the system took the top position, outperforming the organizer baselines by roughly 5–7 points. Absolute scores stay modest (around 0.33–0.36), and the classifier leans toward the majority "retain" class. The work shows that forgetting can be checked from observable signals alone, though the problem remains far from solved.

Key points

  • Results: Task 1 mean scores were 0.3428 (1B) and 0.3565 (3B), against baselines of 0.2813 and 0.2855. Task 2 scores were 0.3345 (1B) and 0.3335 (3B), against baselines of 0.265 and 0.2706.
  • Data: each task has 912 official training items (576 retain, 144 forget, 192 test) and 4,416 test items. A stratified 80/20 split gave 729 training and 183 validation examples. The majority class was not downsampled.
  • Models and source benchmark: four checkpoints were used (model1b_task1_b, model3b_task1_b, model1b_task2_b, model3b_task2_b). They come from the FAME benchmark setting, which uses Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct. Results are reported per unlearning method: advneg, finetune, kl_min, neggrad and pref_opt.
  • Settings: maximum input length was 384 tokens for Task 1 and 512 for Task 2.
  • Error analysis: most confusions fall between the retain and test classes, and this is worse in Task 2, where individual topics rather than whole identities are unlearned. F1 for the forget and test classes is much lower than for retain. Ablations against plain MLPs were not performed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Priyam Saha. 2026. priyam_saha17 at SVELA: A Feature-Centric Pipeline for Verifying Selective Forgetting in Large Language Models. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{saha2026priyam,
  title = {{priyam\_saha17 at SVELA: A Feature-Centric Pipeline for Verifying Selective Forgetting in Large Language Models}},
  author = {Priyam Saha},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.46/}
}

References

12 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Ginart, M. Guan, G. Valiant, J. Zou, Making ai forget you: Data deletion in machine learning, NeurIPS (2019). find
  2. L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, R. Jia, A. Travers, B. Zhang, D. Lie, N. Papernot, Machine unlearning, in: IEEE Symposium on Security and Privacy, 2021. find
  3. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, et al., Extracting training data from large language models, in: USENIX Security Symposium, 2021.
  4. D. Choi, D. Na, Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems, 2023. URL: https://arxiv.org/abs/2311.02240. arXiv:2311.02240 . — [arXiv:2311.02240](https://arxiv.org/abs/2311.02240) ↗
  5. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, J. Z. Kolter, Tofu: A task of fictitious unlearning for llms, 2024. URL: https://arxiv.org/abs/2401.06121. arXiv:2401.06121 . ↗
  6. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, C. Finn, Direct preference optimization: Your language model is secretly a reward model, 2024. URL: https://arxiv.org/abs/2305.18290. arXiv:2305.18290 . ↗
  7. J. Hayes, I. Shumailov, E. Triantafillou, A. Khalifa, N. Papernot, Inexact unlearning needs more careful evaluations to avoid a false sense of privacy, 2024. URL: https://arxiv.org/abs/2403.01218. arXiv:2403.01218 . — [arXiv:2403.01218](https://arxiv.org/abs/2403.01218) ↗
  8. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, Svela at evalita 2026: Overview of the selective verification of erasure from llm answers task, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org,…
  9. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, FAME: Fictional actors for multilingual erasure, in: Proceedings of the Fifteenth Language Resources and Evaluation Conference, European Language Resources Association, 2026.
  10. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… find
  11. D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out-of-distribution examples in neural networks, in: International Conference on Learning Representations, 2017. find
  12. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, 2015. URL: https://arxiv.org/abs/1512.03385. arXiv:1512.03385 . Code and Reproducibility To support transparency and reproducibility, the full implementation, training pipeline, and submission files used for both SVELA t… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saha-2026-priyam-saha17-svela-feature-centric/