Elamur Research⌕
☰
home / papers / 2026 / Grounding Sign Language Representation Learning in Phonology

Grounding Sign Language Representation Learning in Phonology

Paper · 2026 · Toon Vandendriessche, Mathieu De Coster and Joni Dambre

Summary

Most isolated sign language recognition (ISLR) models learn from glosses. Glosses are spoken-language labels that treat each sign as one indivisible unit, so the sub-lexical building blocks of signs stay mixed together in the learned features. The authors instead organise the embedding space around sign phonology, following the Dependency Model developed for NGT and refined for Flemish Sign Language (VGT). Their hierarchical, pose-based network has separate subspaces for handshape and location. It is trained with multi-label phoneme targets for every sequential unit of a sign, not only the first. To ease the scarcity of VGT data, it is trained on large-scale ASL resources and then tested for transfer to VGT.

To check whether phonology is reflected in the geometry of the embeddings, they propose a non-parametric probe that measures how consistent neighbourhoods are as their size grows. Gloss-trained networks hold up only in very small neighbourhoods, which points to surface visual similarity. The disentangled embeddings stay stable at larger scales. This suggests phonological supervision yields more interpretable, robust and cross-lingually transferable representations.

Key points

  • The architecture builds on Demey and van der Kooij's (2008) Dependency Model, in which the sign segment serves as the root unit. It gives handshape and location their own latent subspaces, unlike earlier models (e.g. Kezar et al., 2023b) that used one shared embedding with several classification heads.
  • Inputs are body keypoints from MediaPipe Holistic. This choice reduces noise from video backgrounds and anonymises signers, which helps with GDPR compliance.
  • Phoneme labels come from ASL-LEX 2.0, which covers 2,723 ASL signs. ASL Citizen is listed among the paper's language resources. Unlike prior work, phonemes are predicted for every annotated "morpheme" (sequential unit), not just the first.
  • The new evaluation is a neighbourhood-consistency probe across increasing scales. Under this probe, gloss-based embeddings degrade beyond tiny neighbourhoods, while the phonologically disentangled embeddings remain stable.
  • The authors frame the work as groundwork for phonological dictionary search, computer-assisted language learning, recognition of productive signs, and transfer to low-resource sign languages such as VGT. Movement, orientation and non-manual features are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ghent UniversityGhent · Belgium25 works · 2,949 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 worksHuman-Computer Interaction · 160 works

Cite

Cite as Toon Vandendriessche, Mathieu De Coster and Joni Dambre. 2026. Grounding Sign Language Representation Learning in Phonology. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2kyvv756bfmz

@inproceedings{vandendriessche2026grounding,
  title = {{Grounding Sign Language Representation Learning in Phonology}},
  author = {Toon Vandendriessche, Mathieu {De Coster} and Joni Dambre},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/2kyvv756bfmz},
  url = {https://doi.org/10.63317/2kyvv756bfmz}
}

References

31 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Robbin Battison. 1978. Lexical borrowing in American Sign Language. find
  2. Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al. 2019. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st internat… find
  3. Diane Brentari. 1998. A prosodic model of sign language phonology. MIT Press. find
  4. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10033. find
  5. Hao Chen, Jiaze Wang, Ziyu Guo, Jinpeng Li, Donghao Zhou, Bian Wu, Chenyong Guan, Guangyong Chen, and Pheng-Ann Heng. 2024. SignVTCL: Multi-modal continuous sign language recognition enhanced by visual-textual contrastive learning. arXiv preprint arXiv:2401.11847. — [arXiv:2401.11847](https://arxiv.org/abs/2401.11847) find
  6. Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2024. Machine translation from signed to spoken languages: State of the art and challenges. Universal Access in the Information Society, 23(3):1305–1331. find
  7. Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre. 2020. Sign language recognition with transformer networks. In Proceedings of the twelfth language resources and evaluation conference, pages 6018–6024. find
  8. Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre. 2021. Isolated sign recognition from RGB video using pose flow and self-attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3441–3450. find
  9. Eline Demey. 2005. Fonologie van de Vlaamse Gebarentaal: distinctiviteit en iconiciteit. Ph.D. thesis, Ghent University. find
  10. Eline Demey and Els van der Kooij. 2008. Phonological patterns in a dependency model: Allophonic relations grounded in phonetic and iconic motivation. Lingua, 118(8):1109–1138. find
  11. William Stokoe. 1960. Sign language structure: An outline of the visual communication systems of the American deaf. Studies in Linguistics, Occasional Papers, 8. find
  12. Aashaka Desai, Maartje De Meulder, Julie A Hochgesang, Annemarie Kocab, and Alex X Lu. 2024. Systemic biases in sign language AI research: A deaf-led call to reevaluate research agendas. arXiv preprint arXiv:2403.02563. find
  13. Federico Tavella, Aphrodite Galata, and Angelo Cangelosi. 2022a. Phonology recognition in American Sign Language. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8452–8456. IEEE. find
  14. Ivan Grishchenko and Valentin Bazarevsky. 2020. Mediapipe holistic — simultaneous face, hand and pose prediction, on device. Posted by Research Engineers, Google Research. find
  15. Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, and Alexander H Liu. 2025. SHuBERT: Self-supervised sign language representation learning via multi-stream cluster prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  16. Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu. 2021. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3413– 3423. find
  17. Lee Kezar, Jesse Thomason, and Zed Sehyr. 2023b. Improving sign recognition with phonology. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2732–2737. find
  18. David Laines, Miguel Gonzalez-Mendoza, Gilberto Ochoa-Ruiz, and Gissella Bejarano. 2023. Isolated sign language recognition based on tree structure skeleton images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 276–284. find
  19. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988.
  20. Hamzah Luqman. 2022. An efficient two-stream network for isolated sign language recognition using accumulative video motion. IEEE Access, 10:93785–93798. find
  21. S Renjith, Aneesh Varghese, Manazhy Rashmi, and Poorna SS. 2026. Transformer-based motion-visual integrated fusion for isolated sign language recognition. Computers and Electrical Engineering, 130:110902. find
  22. Federico Tavella, Viktor Schlegel, Marta Romeo, Aphrodite Galata, and Angelo Cangelosi. 2022b. WLASL-LEX: a dataset for recognising phonological properties in American Sign Language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),… find
  23. Els van der Kooij. 2002. Phonological categories in Sign Language of the Netherlands: The role of phonetic implementation and iconicity. Ph.D. thesis, LOT, Utrecht. find
  24. Toon Vandendriessche, Caro Brosens, Hannes De Durpel, Mathieu De Coster, and Joni Dambre. 2026. SignBuddy: From Sign Language Research to Scalable Co-Created Solutions. Universal Access in the Information Society. find
  25. Fangyun Wei and Yutong Chen. 2023. Improving continuous sign language recognition with cross-lingual signs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23612–23621. find
  26. Ronglai Zuo and Brian Mak. 2022. C2SLR: Consistency-enhanced continuous sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5131–5140. 7. find
  27. Language Resource References Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard Ladner, Hal Daumé III, Alex X Lu, Naomi Caselli, et al. 2023. ASL Citizen: a community-sourced dataset for advancing isolated sign language recognition. Advances in Neura… find
  28. Lee Kezar, Jesse Thomason, Naomi Caselli, Zed Sehyr, and Elana Pontecorvo. 2023a. The Sem-Lex Benchmark: Modeling ASL signs and their phonemes. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, pages 1–10. find
  29. Sehyr, Zed Sevcikova and Caselli, Naomi and Cohen-Goldberg, Ariel M and Emmorey, Karen. 2021. The ASL-LEX 2.0 Project: A database of lexical and phonological properties for 2,723 signs in American Sign Language. Oxford University Press. find
  30. Mieke Van Herreweghe, M Vermeerbergen, K De Weerdt, and Katrien Van Mulders. 2004. Woordenboek Nederlands–Vlaamse Gebarentaal/Vlaamse Gebarentaal–Nederlands online. https://woordenboek.vlaamsegebarentaal.be/. ↗
  31. Vlaams Gebarentaalcentrum (VGTC). 2026. Woordenboek Vlaamse Gebarentaal. https://woordenboek.vlaamsegebarentaal.be/search. Accessed: 2026-02-12. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vandendriessche-2026-grounding-sign-language-representation-learning/