Elamur Research⌕
☰
home / papers / 2026 / SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

Paper · 2026 · Noor Islam S. Mohammad and Ulug Bayazit

Summary

The paper asks how one shared encoder can handle structurally different NLP tasks. Three problems get in the way: task interference, class imbalance that distorts feature statistics, and truncation that discards global text statistics. SURGELLM computes a 16-dimensional vector from the full, untruncated input. It holds counts for ten curated lexical indicator groups plus six surface statistics (length, punctuation, digits). The vector enters the model twice: as readable task and feature prefix tokens, and through a learned per-dimension sigmoid gate that mixes it with the [CLS] embedding. Instance-Weighted Normalization (IWN) standardizes the features with class-balanced statistics instead of global ones. This removes majority-class bias and needs no labels at inference. The evaluation covers SST-2, multi-hop retrieval, LLM-prompt attribution and human/LLM authorship detection (17,830 examples, eleven variants, three seeds). The IWN RoBERTa model reaches macro-F1 0.940, and a random-vocabulary control suggests the gains come from lexical content rather than extra parameters. The work positions explicit feature channels as a useful complement to contextual encoders.

Key points

  • SURGELLM-IWN-RoBERTa scores aggregate macro-F1 0.940, which is +0.036 over the strongest non-IWN baseline. IWN alone adds +0.130 F1 on authorship detection, whose corpus is skewed 9.3:1.
  • Backbones are BERT, RoBERTa, DistilBERT and ALBERT, compared against T5-base. Over 99% of parameters are shared across tasks. The ALBERT variant reaches 0.886 avg. F1 with only 11M parameters, and the best model trains about 4× slower than Baseline-DistilBERT.
  • Replacing the vocabulary with random words costs −0.028 avg. F1. An automatic recipe (log-odds with an informative Dirichlet prior, then SBERT + k-means into 10 groups) recovers 99.5% of the performance of manual curation.
  • On the theory side, the paper gives an excess-risk bound tying the gate's benefit to feature alignment. It also proves that the gate degenerates to identity when features carry no information.
  • Main experiments are English-only, with a preliminary French/German test landing within 0.02 F1. Remaining errors involve negation, sarcasm and texts under 30 words. Code and vocabularies are released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Noor Islam S. Mohammad and Ulug Bayazit. 2026. SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.47

@inproceedings{mohammad2026surgellm,
  title = {{SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization}},
  author = {Noor Islam S. Mohammad and Ulug Bayazit},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.47},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.47}
}

References

47 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799–5811. Association for Computa…
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  3. Peter L. Bartlett and Shahar Mendelson. 2002. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482. find
  4. Theophile Blard. 2020. french-sentiment-analysis-with-bert. https://github.com/TheophileBlard/ french-sentiment-analysis-with-bert. ↗
  5. Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41–75. find
  6. Victoria S. Chang, Terri P. Rose, Carol L. Karp, Roy C. Levitt, Constantine Sarantopoulos, and Anat Galor. 2018. Neuropathic-Like Ocular Pain and Nonocular Comorbidities Correlate With Dry Eye Symptoms. Eye & contact lens, 44:S307–S313. find
  7. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357.
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. 2020. CogLTX: Applying BERT to long texts. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 12792–12804. find
  10. Maël Fabien, Esaú Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020. BertAA: BERT fine-tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 127–137. NLP Association of India. find
  11. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  12. Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021). find
  13. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116. Association for Computational Linguistics…
  14. Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1774–1784. Association for Computational Linguistics. find
  15. Zachary Grinberg. 2024. Human vs. LLM text classification corpus. Public dataset release; please update with canonical URL/DOI before camera-ready. Used as the source for task D 4 . Author-check required. find
  16. Michael Crawshaw. 2020. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796. — [arXiv:2009.09796](https://arxiv.org/abs/2009.09796) find
  17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
  18. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9268–9277. find
  19. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 933–941.
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  21. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML).
  22. Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the American Society for Information Science and Technology, 60(1):9–26. find
  23. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR).
  24. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059. Association for Computational Linguistics.
  25. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  26. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. GPT detectors are biased against non-native English writers. Patterns, 4(7):100779.
  27. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980– 2988.
  28. Shikun Liu, Stephen James, Andrew J. Davison, and Edward Johns. 2022. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research (TMLR). find
  29. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496. Association for Computational Linguistics.
  30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  31. LLM-7 Dataset Contributors. 2024. LLM-7: Essays under seven prompt conditions for generation attribution. Public dataset release; please update with canonical reference (URL/DOI) before camera-ready. Cited in this work as “LLM-7 corpus”. Author-check required. find
  32. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 24950–24962.
  33. Burt L. Monroe, Michael P. Colaresi, and Kevin M. Quinn. 2008. Fightin’ Words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  34. Martin Potthast, Francisco Rangel, Michael Tschuggnall, Efstathios Stamatatos, Paolo Rosso, and Benno Stein. 2017. Overview of PAN’17: Author identification, author profiling, and author obfuscation. In Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2017), pages 275–290.… find
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  36. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  37. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  38. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others. 2022. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations (ICLR…
  39. Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018). find
  40. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR). find
  41. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–…
  42. Rupesh K. Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Information Processing Systems 28 (NeurIPS 2015), pages 2377–2385. find
  43. Efstathios Stamatatos. 2009. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology, 60(3):538–556. find
  44. Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 5986–5995. find
  45. Michel Talagrand. 1996. A new look at independence. The Annals of Probability, 24(1):1–34. find
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.…
  47. Sen Wu, Hongyang R. Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944. — [arXiv:2005.00944](https://arxiv.org/abs/2005.00944) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2026-surgellm-rethinking-multi-task-evaluation/