Elamur Research⌕
☰
home / papers / 2026 / Lost in Expression: Diagnosing Systemic Challenges with Non-Manual Generalization in Sign Language Understanding Tasks

Lost in Expression: Diagnosing Systemic Challenges with Non-Manual Generalization in Sign Language Understanding Tasks

Paper · 2026 · Dmitriy Sazonov, Sevgi Z. Gurbuz and Evie A. Malaia

Summary

Sign language models often treat facial and body cues (non-manual markers, NMMs) as secondary, even though these cues carry grammar and meaning. In Saudi Sign Language (SSL), for example, yes/no questions and degree intensification can be signalled without any manual component. The authors argue that score gains from adding facial features are not proof that a model actually uses NMMs in a linguistically faithful way. To examine this, they run a case study on two tasks, Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT). They apply the Uni-Sign model to Isharah-1000 and use its signer-independent and unseen-sentence partitions together with feature attribution to investigate failure cases. The dataset includes veiled signers, whose faces are partly hidden by the Niqab, which makes it a natural test of NMM robustness. The supplied excerpt leaves out the experimental results, so no specific scores are reported here. The work aims to lay groundwork for diagnostic frameworks that test NMM generalization rather than raw benchmark accuracy.

Key points

  • Isharah-1000 contains RGB videos of 18 signers producing up to 1,000 unique SSL sentences, for 15,000 samples in total. For comparison, Phoenix2014-T has 8,257 samples and CSL-Daily has about 21,000.
  • The two controlled splits target different abilities: signer-independent (SI) tests generalization to unseen signers, and unseen sentence (US) tests compositionality.
  • Some Isharah-1000 signers wear the Niqab, which leaves only the eyes visible. This blocks mouth-related cues and may weaken how well head motion can be inferred.
  • Earlier studies found gains of roughly 1.5–13% from adding NMMs. Others found performance drops, both with 70 facial keypoints (Ko et al., 2019) and with only 19 (Johnny et al., 2025), so keypoint count alone does not explain the degradation.
  • The authors stress that attribution maps on their own cannot show faithful NMM use. They call for function-oriented diagnostics and greater involvement of native signers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓North Carolina State UniversityRaleigh · United States13 works · 406 citations🎓University of AlabamaTuscaloosa · United States6 works · 403 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksHuman-Computer Interaction · 141 works

Cite

Cite as Dmitriy Sazonov, Sevgi Z. Gurbuz and Evie A. Malaia. 2026. Lost in Expression: Diagnosing Systemic Challenges with Non-Manual Generalization in Sign Language Understanding Tasks. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ofdq6bwkwik

@inproceedings{sazonov2026lost,
  title = {{Lost in Expression: Diagnosing Systemic Challenges with Non-Manual Generalization in Sign Language Understanding Tasks}},
  author = {Dmitriy Sazonov, Sevgi Z. Gurbuz and Evie A. Malaia},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/3ofdq6bwkwik},
  url = {https://doi.org/10.63317/3ofdq6bwkwik}
}

References

54 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haya Altamimi and Haroon Alsager. 2023. Argument structure and word order in saudi sign language. Journal of Language Teaching and Research, 14:203–214. find
  2. Sarah Alyami and Hamzah Luqman. 2025. Swin-MSTP: Swin transformer with multi-scale temporal perception for continuous sign language recognition. Neurocomputing, 617:129015. find
  3. Sarah Alyami, Hamzah Luqman, Sadam Al-Azani, Maad Alowaifeer, Yazeed Alharbi, and Yaser Alonaizan. 2026. Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition. IEEE Transactions on Multimedia. find
  4. Oya Aran, Thomas Burger, Alice Caplier, and Lale Akarun. 2009. A belief-based sequential fusion approach for fusing manual signs and non-manual signals. Pattern Recognition, 42(5):812– 822. find
  5. Joshua Borneman, Evguenia Malaia, and Ronnie Wilbur. 2018. Motion characterization using optical flow and fractal complexity. Journal of Electronic Imaging, 27:1. find
  6. Sean C. Borneman, Julia Krebs, Ronnie B. Wilbur, and Evie A. Malaia. 2025. Decoding predictive inference in visual language processing via spatiotemporal neural coherence. find
  7. NeurIPS 2025 Workshop on Foundation Models for the Brain and Body. find
  8. Heike Brock, Iva Farag, and Kazuhiro Nakadai. 2020. Recognition of Non-Manual Content in Continuous Japanese Sign Language. Sensors, 20(19):5621. find
  9. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10020– 10030. find
  10. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2021. Multi-channel transformers for multi-articulatory sign language translation. In Computer Vision – ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022.… find
  11. Pedro Alejandro Dal Bianco, Oscar Agustín Stanchi, Facundo Manuel Quiroga, Franco Ronchetti, and Enzo Ferrante. 2024. SignAttention: On the Interpretability of Transformer Models for Sign Language Translation. find
  12. Kenneth DeHaan, Emre Kurtoğlu, Sabyasachi Biswas, Caroline Kobek Pezzarossi, Darrin J. Griffin, Chris Crawford, Ali C. Gurbuz, Evie A. Malaia, Abraham Glasser, Raja Kushalnagar, and Sevgi. Z Gurbuz. 2025. Rf -chesssign: Radar-enabled human-computer interaction in a real- time sign language-controlle… find
  13. A.S. Elons, Menna Ahmed, and Hwaidaa Shedid. 2014. Facial expressions recognition for Arabic Sign Language translation. In 2014 9th International Conference on Computer Engineering & Systems (ICCES), pages 330–335. IEEE. find
  14. Jens Forster, Christoph Schmidt, Oscar Koller, Martin Bellgardt, and Hermann Ney. 2014. Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-weather. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1911–1916, Reykj… find
  15. Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. 2025. SignMusketeers: An efficient multi-stream approach for sign language translation at scale. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22506–22521, Vienna, Austria. Association for Computation… find
  16. Sevgi Z. Gurbuz, Ali C. Gurbuz, Evie A. Malaia, Darrin J. Griffin, Chris Crawford, M. Mahbubur Rahman, Ridvan Aksu, Emre Kurtoglu, Robiul-hossain Mdrafi, Ajaymehul Anbuselvam, Trevor Macks, and Engin Ozcelik. 2020. A linguistic perspective on radar micro-doppler analysis of american sign language. I… find
  17. Aiming Hao, Yuecong Min, and Xilin Chen. 2021. Self-Mutual Distillation Learning for Continuous Sign Language Recognition. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11283–11292. find
  18. Ruth M. Holmes, Ellen Rushe, and Anthony Ventresque. 2024. The key points: Using feature importance to identify shortcomings in sign language recognition models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202… find
  19. Md. Milon Islam, Md Rezwanul Haque, S M Taslim Uddin Raju, and Fakhri Karray. 2025. Fusio-nensemblenet: An attention-based ensemble of spatiotemporal networks for multimodal sign language recognition. In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 4983–4989. tr… find
  20. Julia Krebs, Dietmar Roehm, Ronnie B. Wilbur, and Evie A. Malaia. in press. Visual-Linguistic Cues in Manual Communication: Neural and Behavioral Responses to Mouthings in Sign-Naïve Perception. Brain and Language. Special Issue: Gesture, Cognition and Language. find
  21. Julia Krebs, Ronnie B Wilbur, Dietmar Roehm, and Evie A Malaia. 2025. The interaction of syntax, non-manuals, and prosodic cues as potential topic markers in Austrian Sign Language. Sign Language & Linguistics, 28(1):1–48. find
  22. Emre Kurtoğlu, Kenneth DeHaan, Caroline Kobek Pezzarossi, Darrin J Griffin, Chris Crawford, and Sevgi Z Gurbuz. 2024. Interactive learning of natural sign language with radar. IET Radar, Sonar Navigation, 18(8). find
  23. Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. 2023. Rtmpose: Real-time multi-person pose estimation based on mmpose. find
  24. Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, and Houqiang Li. 2025. Uni-sign: Toward unified sign language understanding at scale. In The Thirteenth International Conference on Learning Representations. find
  25. Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, and Xilin Chen. 2023. Cosign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20619–20629. find
  26. Hamzah Luqman and El-Sayed M. El-Alfy. 2021. Towards Hybrid Multimodal Manual and Non-Manual Arabic Sign Language Recognition: marsl Database and Pilot Study. Electronics, 10(14):1739. find
  27. Samuel Ebimobowei Johnny, Blessed Guda, Andrew Blayama Stephen, and Assane Gueye. 2025. AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition . In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5071–5078, Los Alamitos, CA, USA. IEEE Co… find
  28. Jakub F. Juranek. 2025. Multimodal italian sign language recognition with radar-video late fusion on the multimedalis dataset. In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5079–5085. find
  29. Jung-Ho Kim, Changyong Ko, Mathew Huerta-Enochian, and Seung Yong Ko. 2024. Shedding Light on the Underexplored: Tackling the Minor Sign Language Research Topics. In Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language… find
  30. Sang-Ki Ko, Changjo Kim, Hyedong Jung, and Choongsang Cho. 2019. Neural sign language Hamzah Luqman, Raffaele Mineo, Murtadha Aljubran, Ahmed Abul Hasanaath, Amelia Sorrenti, Sarah Alyami, Sadam Al-Azani, Maad Alowaifeer, JiHwan Moon, Vaclav Javorek, Tomas Zelezny, Marek Hruz, Gaia Caligiore, Silvio… find
  31. The SignEval 2025 Challenge at the ICCV Multimodal Sign Language Recognition Workshop: Results and Discussion . In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5086–5095, Los Alamitos, CA, USA. IEEE Computer Society. find
  32. Evie Malaia, Joshua D Borneman, and Ronnie B Wilbur. 2018. Information transfer capacity of articulators in American Sign Language. Language and speech, 61(1):97–112. find
  33. Evie A. Malaia, Joshua Borneman, and Sevgi Gurbuz. 2024. Capturing motion: Using radar to build better sign language corpora. In Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources, pages 384–389, Torino, Ita… find
  34. Evie A Malaia, Sean C Borneman, Joshua D Borneman, Julia Krebs, and Ronnie B Wilbur. 2023. Prediction underlying comprehension of human motion: an analysis of Deaf signer and non-signer EEG in response to visual stimuli. Frontiers in Neuroscience, 17:1218510. find
  35. Evie A Malaia, Sean C Borneman, Julia Krebs, and Ronnie B Wilbur. 2021. Low-frequency entrainment to visual motion underlies sign language comprehension. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:2456–2463. find
  36. Evie A Malaia and Ronnie B Wilbur. 2020. Syllable as a unit of information transfer in linguistic communication: The entropy syllable parsing model. Wiley Interdisciplinary Reviews: Cognitive Science, 11(1):e1518. find
  37. Sultan Mohammad Manjur, Sabyasachi Biswas, and Ali C. Gurbuz. 2025. A multimodal video and radar fusion framework for high-accuracy isolated sign language recognition. In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5061– 5070. find
  38. Kenneth Mejía-Peréz, Diana-Margarita Córdova-Esparza, Juan Terven, Ana-Marcela Herrera-Navarro, Teresa García-Ramírez, and Alfonso Ramírez-Pedraza. 2022. Automatic Recognition of Mexican Sign Language Using a Depth Camera and Recurrent Neural Networks. Applied Sciences, 12(11):5523. find
  39. Carla Morris and Erin Schneider. 2012. On selected morphemes in saudi arabian sign language. Sign Language Studies, 13:103–121. find
  40. Medet Mukushev, Arman Sabyrov, Alfarabi Imashev, Kenessary Koishybay, Vadim Kimmelman, and Anara Sandygulova. 2020. Evaluation of manual and non-manual components for sign language recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6073–6078, Marseille, Fr… find
  41. Dinh Nam Pham and Eleftherios Avramidis. 2025. The importance of facial features in vision-based sign language recognition: Eyes, mouth or full face? In Adjunct Proceedings of the 25th ACM International Conference on Intelligent Virtual Agents, IVA Adjunct ’25, New York, NY, USA. Association for Com… find
  42. Sylvie Ong and Surendra Ranganath. 2005. Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE transactions on pattern analysis and machine intelligence, 27:873–91. find
  43. Junfu Pu, Wengang Zhou, and Houqiang Li. 2016. Sign language recognition with multi-modal features. In Advances in Multimedia Information Processing - PCM 2016, pages 252–261, Cham. Springer International Publishing. find
  44. Dmitriy Sazonov, Kamrul Islam, Evie Malaia, and Sevgi Z. Gurbuz. 2025. Modality-specific benchmarks and radar range-doppler envelope classification for multimodal isolated sign language recognition. In 2025 IEEE/CVF International Conference on Computer Vision Workshops (IC-CVW), pages 5046–5053. find
  45. Ozge Mercanoglu Sincan, Julio C. S. Jacques Junior, Sergio Escalera, and Hacer Yalim Keles. 2021. ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research. find
  46. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR. find
  47. Garrett Tanzer, Maximus Shengelia, Ken Harren-stien, and David Uthus. 2024. Reconsidering sentence-level sign language translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6262–6287, Miami, Florida, USA. Association for Computational Linguisti… find
  48. Ulrich von Agris, Moritz Knorr, and Karl-Friedrich Kraiss. 2008. The significance of facial features for automatic sign language recognition. In 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pages 1–6. IEEE. find
  49. Ronnie B Wilbur. 2021. Non-manual markers: Theoretical and experimental perspectives. In The Routledge handbook of theoretical and experimental sign language research, pages 530–565. Routledge. find
  50. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  51. Tomas Zelezny, Jakub Straka, Vaclav Javorek, Ondrej Valach, Marek Hruz, and Ivan Gruber. 2025. Exploring pose-based sign language translation: Ablation studies and attention insights. find
  52. Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. 2023. Gloss-free sign language translation: Improving from visual-language pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20814– 20824. find
  53. Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021. Improving sign language translation with monolingual data by sign back-translation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1316–1325. 10. find
  54. Language Resource References Sarah Alyami, Hamzah Luqman, Sadam Al-Azani, Maad Alowaifeer, Yazeed Alharbi, and Yaser Alonaizan. 2026. Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition. PID https://doi.org/10.1109/TMM.2026.3664959. — [doi:10.1109/tmm.2026.3664959](https://doi.org/10.1109/tmm.2026.3664959) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sazonov-2026-lost-expression-diagnosing-systemic-challenges/