Elamur Research⌕
☰
home / papers / 2026 / Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models

Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models

Paper · 2026 · Sumanth Manduru and Carlotta Domeniconi

Summary

Small Language Models (SLMs) are increasingly deployed on edge devices, yet fairness audits have mostly targeted either very large LLMs (8B+) or encoder-scale models such as BERT, leaving the 0.5–5B range largely unexamined. The authors argue that a low bias score alone can mislead: a model may seem neutral only because it guesses randomly, abstains indiscriminately, or follows shallow heuristics. They call this failure mode "vacuous neutrality." To detect it, they propose VaNeu, a four-stage audit:

  1. Bias.
  2. Utility, i.e. task correctness.
  3. Ambiguity handling, i.e. whether "Unknown" is chosen appropriately.
  4. Positional bias, i.e. preference for particular answer slots.

They apply it to nine instruction-tuned SLMs from four families, mainly on BBQ in both ambiguous and disambiguated contexts, and compare the results with StereoSet and CrowS-Pairs. Models that score well on bias often fail the later stages. Bias-only benchmarks also cannot separate genuine fairness from degenerate behavior. The work matters for anyone choosing compact models for socially sensitive applications, because it shows that fairness checks need several dimensions.

Key points

  • Models audited: nine instruction-tuned SLMs from the Qwen2.5, LLaMA3.2, Gemma3 and Phi families, all in the 0.5B–5B range. The authors present this as the first comparative fairness audit across several families at this scale.
  • Metrics for each stage:
    • Bias: bias score.
    • Utility: F1.
    • Ambiguity handling: Target-to-NonTarget Ratio (TNR) and Unknown Ratio (UR).
    • Positional bias: normalized KL divergence over answer positions A, B and C.
  • Bias-only vs. full evaluation: On StereoSet and CrowS-Pairs, LLaMA-3.2-3B, Gemma3-4B, Phi-3.5-mini and Phi-4-mini all look acceptable. Under BBQ with VaNeu they diverge sharply:
    • LLaMA-3.2-3B and Gemma3-4B prove brittle.
    • Phi-4-mini is the most robust.
    • Phi-3.5-mini falls in between.
  • Category differences:
    • Age and Disability Status: the smallest variants tend toward stereotypes, while larger ones lean mildly anti-stereotypical.
    • Race and Sexual Orientation: bias stays low, with strong utility.
  • Limitations and code: the datasets are U.S.-centric and in QA format. The authors plan a composite single-score metric. Code is on GitHub (smanduru10/Vacuous-Neutrality-Framework).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Sumanth Manduru and Carlotta Domeniconi. 2026. Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.52

@inproceedings{manduru2026beyond,
  title = {{Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models}},
  author = {Sumanth Manduru and Carlotta Domeniconi},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.52},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.52}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. 2023. Reducing the carbon impact of generative ai inference (today and in 2035). In Proceedings of the 2nd Workshop on Sustainable Computer Systems, HotCarbon ’23, New York, NY, USA. Association for Comp… find
  3. Gustavo Gonçalves and Emma Strubell. 2023. Understanding the effect of model compression on social bias in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2663–2675, Singapore. Association for Computational Linguistics. find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  5. Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, and Chandan K. Reddy. 2025. Mitigating selection bias with node pruning and auxiliary options. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5190–5215, Vienna, Austria.… find
  6. Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. 2024. Decoding compressed trust: Scrutinizing the trustworthiness of efficient ll… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  8. Yue Huang, Qihui Zhang, Philip S. Y, and Lichao Sun. 2023. Trustgpt: A benchmark for trustworthy and responsible large language models. Preprint, arXiv:2306.11507. — [arXiv:2306.11507](https://arxiv.org/abs/2306.11507) find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.
  10. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489, Online. Association for Computational Linguistics.
  11. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. Preprint, arXiv:2301.00774. find
  12. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024a. Bias and fairness in large language models: A survey. Preprint, arXiv:2309.00770.
  13. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. 2024b. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. Preprint, arXiv:2402.01981. find
  14. GemmaTeam, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev… find
  15. Masahiro Kaneko and Danushka Bollegala. 2021. Unmasking the mask – evaluating social biases in masked language models. Preprint, arXiv:2104.07496. — [arXiv:2104.07496](https://arxiv.org/abs/2104.07496) find
  16. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Y. Wang. 2023. A survey on fairness in large language models. ArXiv, abs/2308.10149. — [arXiv:2308.10149](https://arxiv.org/abs/2308.10149) find
  17. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024a. Awq: Activation-aware weight quantization for llm compression and acceleration. Preprint, arXiv:2306.00978. find
  18. Yi-Cheng Lin, Tzu-Quan Lin, Hsi-Che Lin, Andy T. Liu, and Hung-yi Lee. 2024b. On the social bias of speech self-supervised models. In Interspeech 2024, interspeech 2024, page 4638–4642. ISCA. find
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  20. Llama3.2. 2024. Llama 3.2 connect: 2024 vision for edge and mobile devices. Accessed: 2025-05-07. find
  21. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. Preprint, arXiv:2004.09456.
  22. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistic…
  23. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  24. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  25. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  27. Krithika Ramesh, Arnav Chavan, Shrey Pandit, and Sunayana Sitaram. 2023. A comparative study on the impact of model compression techniques on fairness in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15762–… find
  28. Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. 2024. Addressing cognitive bias in medical language models. Preprint, arXiv:2402.08113. — [arXiv:2402.08113](https://arxiv.org/abs/2402.08113) find
  29. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. Preprint, arXiv:2306.11695. find
  30. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  31. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. Preprint, arXiv:2304.06588.
  32. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao ’Kenneth’ Huang, and Shomir Wilson. 2023. Nationality bias in text generation. Preprint, arXiv:2302.02463. find
  33. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5598– 5621, Bangkok, Thailand. Association for Computation… find
  34. Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, and Vivek Srikumar. 2024. Beyond perplexity: Multidimensional safety evaluation of llm compression. Preprint, arXiv:2407.04965. — [arXiv:2407.04965](https://arxiv.org/abs/2407.04965) find
  35. Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A survey on model compression for large language models. Preprint, arXiv:2308.07633.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/manduru-2026-beyond-bias-scores-unmasking-vacuous/