Elamur Research⌕
☰
home / papers / 2026 / Continuous Sign Language Recognition using Multimodal Input and Handshape-aware Boundary Detection

Continuous Sign Language Recognition using Multimodal Input and Handshape-aware Boundary Detection

Paper · 2026 · Mingyu Zhao, Zhanfu Yang, Yang Zhou et al.

Summary

Recognizing signs in continuous American Sign Language (ASL) is harder than classifying isolated clips. Each sign's start and end frames must be found, and signs in running discourse differ from their dictionary (citation) forms. The authors use a two-step pipeline: they first split a sentence video into signs, then classify each segment. Segmentation is framed as frame-level BIO tagging over 3D skeletal features. A spatio-temporal convolutional network processes these features, with added velocity and acceleration cues. Certain handshapes tend to appear where signs begin and end. To exploit this, the authors pretrain a classifier for 87 linguistically defined handshape categories on a dataset merged and normalized from several existing sources. Its features are fed into the segmentation stream through gated cross-attention. The recognizer is trained on both citation-form signs and signs cut from continuous signing using manual annotations. On the ASLLRP corpus, the system reportedly outperforms earlier approaches by a clear margin. This matters because reliable boundary detection is a prerequisite for practical CSLR applications such as captioning or interpretation support.

Key points

  • Segmentation uses B/I/O labels per frame rather than plain IO tagging. This lets the model separate signs that directly follow each other with no "outside" frames between them.
  • The handshape classifier covers 87 canonical handshape categories. It is pretrained on a newly unified dataset built from several existing collections, then fused with the segmentation model through cross-attention with a gating mechanism.
  • Evaluation is done on ASLLRP data, with gloss labels kept consistent across all datasets used. Recognition is applied only to signs with at least 6 training examples, so the system is not yet fully end-to-end.
  • The scope is limited to right-handed signers and signs involving the dominant hand. Fingerspelling, classifiers and gestures are not handled separately.
  • The available text claims significant gains over previous work, but the excerpt omits the exact scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rutgers, The State University of New JerseyNew Brunswick · United States40 works · 12,486 citations🎓Boston UniversityBoston · United States30 works · 4,343 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 worksHuman-Computer Interaction · 160 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Shuhang Lin, Carol Neidle and Dimitri Metaxas. 2026. Continuous Sign Language Recognition using Multimodal Input and Handshape-aware Boundary Detection. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/22bkv35eirvm

@inproceedings{zhao2026continuous,
  title = {{Continuous Sign Language Recognition using Multimodal Input and Handshape-aware Boundary Detection}},
  author = {Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Shuhang Lin, Carol Neidle and Dimitri Metaxas},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/22bkv35eirvm},
  url = {https://doi.org/10.63317/22bkv35eirvm}
}

References

52 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ASL-deafined. 2017. Deaf and Hard of Hearing Population in the United States (expanded overview). https://blog.asldeafined.com/2025/ 04/deaf-and-hard-of-hearing-popul ation-in-the-united-states-expande d-overview/. ↗
  2. Matyáš Boháček and Marek Hrúz. 2022. Sign pose-based transformer for word-level sign language recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 182–191. find
  3. Diane Brentari. 2019. Sign Language Phonology. University of Chicago Press. find
  4. Hannah Bull, Triantafyllos Afouras, Gül Varol, Samuel Albanie, Liliane Momeni, and Andrew Zisserman. 2021. Aligning subtitles in sign language videos. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11552–11561. find
  5. Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, and Richard Bowden. 2017. Subunets: End-to-end hand shape and continuous sign language recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3075–3084. find
  6. Konstantinos M. Dafnis, Evgenia Chroni, Carol Neidle, and Dimitri Metaxas. 2022. Isolated sign recognition using ASL datasets with consistent text-based gloss labeling and curriculum learning. In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Ju… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. 2023. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7157–7173. find
  9. Iva Farag and Heike Brock. 2019. Learning motion disfluencies for automatic sign language segmentation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7360–7364. find
  10. Jens Forster, Christoph Schmidt, Oscar Koller, Martin Bellgardt, and Hermann Ney. 2014. Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-weather. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1911–1916, Reykj… find
  11. Matilde Gonzalez and Christophe Collet. 2011. Sign segmentation using dynamics and hand configuration for semi-automatic annotation of sign language corpora. In International Gesture Workshop, pages 204–215. Springer. find
  12. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, pages 369–376. find
  13. Leming Guo, Wanli Xue, Qing Guo, Bo Liu, Kaihua Zhang, Tiantian Yuan, and Shengyong Chen. 2023. Distilling cross-temporal contexts for continuous sign language recognition. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 10771– 10780. find
  14. Leming Guo, Wanli Xue, Bo Liu, Kaihua Zhang, Tiantian Yuan, and Dimitris Metaxas. 2024. Gloss prior guided visual feature learning for continuous sign language recognition. IEEE Transactions on Image Processing, 33:3486–3495. find
  15. Haoyu Han, Mengdi Zhang, Min Hou, Fuzheng Zhang, Zhongyuan Wang, Enhong Chen, Hongwei Wang, Jianhui Ma, and Qi Liu. 2020. Stgcn: A spatial-temporal aware graph learning method for poi recommendation. In 2020 IEEE International Conference on Data Mining (ICDM), pages 1052–1057. find
  16. Low Jian He, Harry Walsh, Ozge Mercanoglu Sincan, and Richard Bowden. 2025. Hands-on: Segmenting individual signs from continuous sequences. In 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–5. IEEE. find
  17. Jiani Huang, Wengang Zhou, Houqiang Zhang, and Weiping Li. 2018. Video-based sign language recognition without temporal segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 2257–2264. Article No. 275. find
  18. Zifan Jiang, Gerard Sant, Amit Moryossef, Mathias Müller, Rico Sennrich, and Sarah Ebling. 2024. SignCLIP: Connecting text and sign language by contrastive learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9171–9193, Miami, Florida, USA. Assoc… find
  19. Can Jin, Ying Li, Mingyu Zhao, Shiyu Zhao, Zhenting Wang, Xiaoxiao He, Ligong Han, Tong Che, and Dimitris N. Metaxas. 2025. Lor-VP: Low-rank visual prompting for efficient vision model adaptation. In The Thirteenth International Conference on Learning Representations, pages 9359–9376. find
  20. Tonni Das Jui, Gissella Bejarano, and Pablo Rivas. 2022. A machine learning-based segmentation approach for measuring similarity between sign languages. In Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources, pages 94… find
  21. Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on Applications of Computer Vision, pages 1459–1469. find
  22. Natalia Lusin, Terri Peterson, Christine Sulewski, and Rizwana Zafer. 2023. Enrollments in languages other than English in US institutions of higher education, fall 2021 - Modern Language Association of America. find
  23. Amit Moryossef, Zifan Jiang, Mathias Müller, Sarah Ebling, and Yoav Goldberg. 2023. Linguistically motivated sign language segmentation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12703–12724, Singapore. Association for Computational Linguistics. find
  24. Boston University (BU) National Center for Sign Language and Gesture Resources. 2012. NC-SLGR Handshapes. Videos available from https://www.bu.edu/asllrp/cslgr/page s/ncslgr-handshapes.html. ↗
  25. Carol Neidle. 2023. Challenges for linguistically-driven computer-based sign recognition from continuous signing for American Sign Language. arXiv preprint arXiv:2311.00762. — [arXiv:2311.00762](https://arxiv.org/abs/2311.00762) find
  26. Carol Neidle and Carey Ballard. 2022a. Revised Gloss Labels for Signs from the WLASL Dataset - Boston University. http://www.bu.edu/a sllrp/wlasl-alt-glosses.pdf. ↗
  27. Carol Neidle and Carey Ballard. 2022b. Why alternative gloss labels will increase the value of the WLASL dataset - ASLLRP Project Report no. 21, Boston University. Project reports available from https://www.bu.edu/asllrp/repo rts.html. ↗
  28. Carol Neidle and Dimitris Metaxas. 2024a. American Sign Language Linguistic Research Project (ASLLRP) Sign Bank: Boston University American Sign Language Lexicon Video Dataset (ASLLVD) version 8 - June. Videos and annotations available from https://dai.cs.rutge rs.edu/dai/s/signbank. ↗
  29. Carol Neidle and Dimitris Metaxas. 2024b. American Sign Language Linguistic Research Project (ASLLRP) Sign Bank: DawnSignPress (DSP) ASL Sign Collection, version 6. Videos provided by DawnSignPress. find
  30. Carol Neidle and Dimitris Metaxas. 2024c. American Sign Language Linguistic Research Project (ASLLRP) Sign Bank: Rochester Institute of Technology (RIT) collection, version 6. Videos and annotations available from https://dai.cs.rutgers.edu/dai/s/signbank. ↗
  31. Carol Neidle and Dimitris Metaxas. 2025a. American Sign Language Linguistic Research Project (ASLLRP) Sign Bank: ASLLRP Continuous Signing Corpora - version 3 - June 2025. Videos and annotations available from https://dai.cs.rutgers.edu/dai/s/signbank. ↗
  32. Carol Neidle and Dimitris Metaxas. 2025b. American Sign Language Linguistic Research Project (ASLLRP) Sign Bank: DawnSignPress (DSP) Sentences Collection, version 4. Videos provided by DawnSignPress. find
  33. Carol Neidle and Augustine Opoku. 2022. Addendum to: Documentation for Download of ASLLRP Sign Bank Citation-form Sign Datasets - ASLLRP Project Report no. 20, Boston University. Project reports available from https://www.bu.edu/asllrp/reports.html. ↗
  34. Carol Neidle, Augustine Opoku, Carey Ballard, Konstantinos M Dafnis, Evgenia Chroni, and Dimitris Metaxas. 2022a. Resources for computer-based sign recognition from video, and the criticality of consistency of gloss labeling across multiple large ASL video corpora. In Proceedings of the LREC2022 10t… find
  35. Carol Neidle, Augustine Opoku, and Dimitris Metaxas. 2022b. ASL Video Corpora & a Sign Bank: Resources available through the American Sign Language Linguistic Research Project (ASLLRP). arXiv preprint arXiv:2201.07899. — [arXiv:2201.07899](https://arxiv.org/abs/2201.07899) find
  36. Carol Neidle, Ashwin Thangali, and Stan Sclaroff. 2012. Challenges in development of the American Sign Language Lexicon Video Dataset (ASLLVD) corpus. In Proceedings of the LREC2012 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, pages 14… find
  37. Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, and Benjamin Schrauwen. 2015. Sign language recognition using convolutional neural networks. In Computer Vision – ECCV 2014 Workshops, pages 572–578. Springer. find
  38. Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2024. A transformer model for boundary detection in continuous sign language. Multimedia Tools and Applications, 83(42):89931– 89948. find
  39. Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2025. A non-anatomical graph structure for boundary detection in continuous sign language. Scientific Reports, 15(1):25683. find
  40. Katrin Renz, Nicolaj C Stache, Samuel Albanie, and Gül Varol. 2021. Sign language segmentation with temporal convolutional networks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2135–2139. find
  41. Adam Schembri, Jordan Fenlon, Ramas Rentelis, Sally Reynolds, and Kearsy Cormier. 2013. Building the British Sign Language corpus. Language Documentation & Conservation, 7:136–154. find
  42. Thad Starner and Alex Pentland. 1998. Real-time American Sign Language recognition using desk and wearable computer-based video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1371–1375. find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000– 6010.
  44. Christian Vogler and Dimitris Metaxas. 1999. Parallel Hidden Markov Models for American Sign Language Recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 116–122. find
  45. Christian Vogler and Dimitris Metaxas. 2001. A framework for recognizing the simultaneous aspects of American Sign Language. Computer Vision and Image Understanding, 81(3):358–384. find
  46. Polina Yanovich, Carol Neidle, and Dimitris Metaxas. 2016. Detection of major ASL sign types in continuous signing for ASL recognition. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3067–3073. find
  47. Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, and De Hu. 2023. C2st: Cross-modal contextualized sequence transduction for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21053–21062. find
  48. Sen Zhang, Xiaoxiao He, Di Liu, Zhaoyang Xia, Mingyu Zhao, Chaowei Tan, Vivian Li, Bo Liu, Dimitris N. Metaxas, and Mubbasir Kapadia. 2026. Large sign language models: Toward 3D American Sign Language translation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W… find
  49. Xuan Zhang and Kevin Duh. 2023. Handshape-aware sign language recognition: Extended datasets and exploration of handshape-inclusive methods. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2993–3002. find
  50. Yan Zhang, Wanli Xue, Yuxi Zhou, Tiantian Yuan, and Shengyong Chen. 2025. Core: multi-link graph attention network with inter-regional collaboration for continuous sign language recognition. Pattern Recognition, 167:111716. find
  51. Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2020. Spatial-temporal multi-cue network for continuous sign language recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13009– 13016. find
  52. Yang Zhou, Zhaoyang Xia, Yuxiao Chen, Carol Neidle, and Dimitris N. Metaxas. 2024. A multimodal spatio-temporal GCN model with enhancements for isolated sign recognition. In Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign L… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-continuous-sign-language-recognition-multimodal/