Elamur Research⌕
☰
home / papers / 2026 / A Video-Based Reverse Dictionary for Sign Language Using Gesture Similarity

A Video-Based Reverse Dictionary for Sign Language Using Gesture Similarity

Paper · 2026 · Batyrbek Orazumbekov, Daniyal Bayanov, Aruzhan Kaltay et al.

Summary

Most sign language recognition work treats the task as classification into a fixed gloss inventory. That does not help someone who has seen a sign but cannot name it, and so cannot type a text query. The authors build a retrieval system for isolated signs instead. A query video is mapped into an embedding space, and the closest stored videos are returned, giving a reverse dictionary driven only by motion. The input is skeletal keypoints rather than RGB frames, which reduces sensitivity to background and signer appearance. Two temporal encoders are compared: a self-attention Transformer and an ST-GCN. Their embeddings are shaped with supervised contrastive learning and ArcFace loss. Evaluation on WLASL, where each gloss has very few examples, uses the ranking metrics Recall@K and mAP. In this low-shot setting the Transformer outperforms the graph model, and attention-based pooling over time adds further gains. The best model reaches an mAP of 0.237. Applied to AUTSL without retraining, the model still transfers meaningfully, which suggests that text-free visual sign search is practical.

Key points

  • Task: pose-based video-to-video retrieval of isolated signs (a reverse gesture dictionary), with no textual or gloss-alignment supervision at query time.
  • Architectures: Transformer encoder vs. ST-GCN. Metric learning losses: the supervised contrastive objective introduced by Khosla et al. (2020), plus ArcFace, which enforces an angular margin.
  • Data: trained on WLASL, which averages only about 3–4 samples per gloss. Best result is mAP 0.237 on the WLASL validation set, obtained with Transformer plus attention-based temporal pooling.
  • Generalization: the WLASL-trained model was tested on a 226-label AUTSL subset of unseen labels and gave non-trivial retrieval performance.
  • Future work (Nazarbayev University team):
    • scaling search with FAISS/HNSW indexing;
    • cross-lingual tests on internal Russian and Kazakh sign datasets;
    • extending from isolated signs to continuous sequences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nazarbayev UniversityAstana · Kazakhstan4 works · 44 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksHuman-Computer Interaction · 141 works

Cite

Cite as Batyrbek Orazumbekov, Daniyal Bayanov, Aruzhan Kaltay and Anara Sandygulova. 2026. A Video-Based Reverse Dictionary for Sign Language Using Gesture Similarity. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/54ysrywg3ktr

@inproceedings{orazumbekov2026video,
  title = {{A Video-Based Reverse Dictionary for Sign Language Using Gesture Similarity}},
  author = {Batyrbek Orazumbekov, Daniyal Bayanov, Aruzhan Kaltay and Anara Sandygulova},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/54ysrywg3ktr},
  url = {https://doi.org/10.63317/54ysrywg3ktr}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fatih Cakir, Kaiming He, Xunxia Xia, Brian Kulis, and Stan Sclaroff. 2019. Deep metric learning to rank. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1861–1870. find
  2. Yiting Cheng, Fangyun Wei, Jianmin Bao, Dong Chen, and Wenqiang Zhang. 2023. CiCo: Domain-aware sign language retrieval via cross-lingual contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19016–19026. find
  3. Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. ArcFace: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699. video-based American Sign Language dictionary: Exploring design a… find
  4. Zifan Jiang, Gerard Sant Muniesa, Amit Moryossef, Mathias Müller, Rico Sennrich, and Sarah Ebling. 2024. SignCLIP: Connecting text and sign language by contrastive learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 9171–9193. find
  5. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661–18673. find
  6. Dongxu Li, Cristian Rodriguez Opazo, Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1459– 1469. find
  7. Xiang Li, Xinyu Yang, Zhen Ma, and Ji-Hong Xue. 2023. Deep metric learning for few-shot image classification: A review of recent developments. Pattern Recognition, 138:109381. find
  8. Yury A. Malkov and Dmitry A. Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836.
  9. Yueqi Duan, Jiwen Lu, Wenzhao Zheng, and Jie Zhou. 2020. Deep adversarial metric learning. IEEE Transactions on Image Processing, 29:2037–2051. find
  10. Gonçalo Vinagre Martins. 2024. SLVideo: A sign language video moment retrieval framework. Master’s thesis, Universidade NOVA de Lisboa (Portugal). find
  11. Amanda Duarte, Samuel Albanie, Xavier Giró i Nieto, and Gül Varol. 2022. Sign language video retrieval with free-form textual queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14094– 14104. find
  12. Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. 2020. A metric learning reality check. In Proceedings of the European Conference on Computer Vision, pages 681–699. find
  13. Simeon Emanuilov and Aleksandar Dimov. 2024. Billion-scale similarity search using a hybrid indexing approach with advanced filtering. Cybernetics and Information Technologies, 24(4):45– 58. find
  14. Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. 2022. Spectral, probabilistic, and deep metric learning: Tutorial and survey. arXiv preprint arXiv:2201.09267. — [arXiv:2201.09267](https://arxiv.org/abs/2201.09267) find
  15. Saad Hassan, Matyáš Boháček, Chaelin Kim, and Denise Crochet. 2025. Towards an AI-driven Mahnaz Parian-Scherb, Peter Uhrig, Luca Rossetto, Stéphane Dupont, and Heiko Schuldt. 2024. Gesture retrieval and its application to the study of multimodal communication. International Journal on Digital Librar… find
  16. Vignesh Ramanathan, Kevin Tang, Greg Mori, and Li Fei-Fei. 2015. Learning temporal embeddings for complex video analysis. In Proceedings of the IEEE International Conference on Computer Vision, pages 4471–4479. find
  17. Asmat Zahra, Nazia Perwaiz, Muhammad Shahzad, and Muhammad Moazam Fraz. 2023. Person re-identification: A retrospective on domain-specific open challenges and future trends. Pattern Recognition, 142:109669. 9. find
  18. Language Resource References Eleni Efthimiou and Stavroula-Evita Fotinea and Thomas Hanke and John Glauert and Richard Bowden and Annelies Braffort and Christophe Collet and Petros Maragos and François Lefebvre-Albaret. 2012. Dicta-Sign Wiki. Dicta-Sign Project. https://doi.org/10.1007/ 978-3-642-31… find
  19. Dongxu Li and Cristian Rodriguez Opazo and Xin Yu and Hongdong Li. 2020. WLASL: Word-Level American Sign Language Dataset. Australian National University. https://doi.org/10. 1109/WACV45572.2020.9093512. find
  20. Ozge Mercanoglu Sincan and Hacer Yalim Keles. 2020. AUTSL: Ankara University Turkish Sign Language Dataset. Ankara University. https://doi.org/10.1109/ACCESS. 2020.3028072. — [doi:10.1109/access](https://doi.org/10.1109/access) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/orazumbekov-2026-video-based-reverse-dictionary-sign/