Elamur Research⌕
☰
home / papers / 2026 / Generating Sign Language Poses from HamNoSys and Natural Language Descriptions

Generating Sign Language Poses from HamNoSys and Natural Language Descriptions

Summary

The paper addresses the middle stage of a three-step sign language generation pipeline: turning a sign transcribed in HamNoSys into a sequence of body, hand and face keypoints. The authors build on Ham2Pose, which trained a transformer from scratch. They ask whether a pretrained LLM, adapted as MotionGPT does for general human motion, can do this job better. Because LLMs have seen almost no HamNoSys, the symbols are first rendered as English text. The authors tried several renderings: raw symbol code names, two hand-built symbol dictionaries, and fluent paraphrases produced by an LLM. An instruction-tuned LLM then outputs discrete pose codes, and a VQ-VAE decodes these codes back into keypoint sequences. Experiments use the multilingual Ham2Pose dataset and its nDTW-MJE-based metrics. Even plain dictionary-style descriptions improve pose prediction, which shows that pretrained language knowledge can carry over to sign phonology. Hand configuration remains the weakest part of the output. The authors also release a HamNoSys-to-text dictionary as a resource.

Key points

  • Dataset: Same data as Ham2Pose, about 5,700 sign videos with HamNoSys transcriptions in four sign languages: PJM (2568), DGS (1911), GSL (889) and LSF (381). Sources are the DGS Corpus, Dicta-Sign and a PJM corpus-based dictionary. The data is split roughly 81/9/10 into train, dev and test.
  • Pose representation: 137 OpenPose keypoints in 2D, giving a 274-dimensional input. They are discretized with a VQ-VAE (codebook of 512 codes, based on the MotionGPT/T2M-GPT implementation) using a downsampling rate of 2.
  • VQ-VAE design choice: A minimum sequence length of 32 was chosen over 64 because it keeps 94.5% of the training data instead of about 40%. On the test set it reaches a reconstruction loss of 0.00168 and nDTW-MJE of 7.99.
  • Evaluation: nDTW-MJE, a DTW-based joint error that handles missing keypoints, plus prediction and ground-truth distance ranks against 41 candidate sequences.
  • Limitations and future work: Hand shapes are reproduced poorly, and only isolated signs are handled, not full sentences. Proposed next steps include separate hand codebooks and a hand-weighted metric.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/466di7tv7dpd

@inproceedings{maximo2026generating,
  title = {{Generating Sign Language Poses from HamNoSys and Natural Language Descriptions}},
  author = {Máximo, Santiago and Chiruzzo, Luis},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/466di7tv7dpd},
  url = {https://doi.org/10.63317/466di7tv7dpd}
}

References

25 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne E Baker. 2016. Sign languages as natural languages. In The linguistics of sign languages: An introduction, pages 1–24. John Benjamins Publishing Company. find
  2. J. Andrew Bangham, Stephen Cox, Ralph Elliott, John R. W. Glauert, Ian Marshall, Siniša Rankov, and Mark Wells. 2000. Virtual signing: capture, animation, storage and transmission-an overview of the ViSiCAST project. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in neural information processing systems, 33:1877–1901. find
  4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. Advances in neural information processing systems, 36:10088–10115. find
  5. L Goyal. 2015. Review and Comparison of Writing Notations of Sign Language. International Journal of Engineering Sciences (IJoES). find
  6. Thomas Hanke. 2004. HamNoSys–representing sign language data in language resources and language processing contexts. In sign-lang@ LREC 2004, pages 1–6. European Language Resources Association (ELRA). find
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 1(2):3. find
  8. Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian. 2021. Towards fast and high-quality sign language production. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3172–3181. find
  9. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lac… find
  10. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-Source Toolkit for Neural Machine Translation. In Annual Meeting of the Association for Computational Linguistics. find
  11. Ben Saunders, Necati Cihan Camgöz, and R. Bowden. 2020a. Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video. ArXiv, abs/2011.09846. — [arXiv:2011.09846](https://arxiv.org/abs/2011.09846) find
  12. Ben Saunders, Necati Cihan Camgöz, and R. Bowden. 2020b. Progressive Transformers for End-to-End Sign Language Production. ArXiv, abs/2004.14874. — [arXiv:2004.14874](https://arxiv.org/abs/2004.14874) find
  13. Rotem Shalev-Arkushin, Amit Moryossef, and Ohad Fried. 2023. Ham2pose: Animating sign language notation into pose sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21046–21056. find
  14. Robert Smith. 2013. HamNoSys 4.0 User guide. find
  15. Stephanie Stoll, Necati Cihan Camgoz, Simon Hadfield, and R. Bowden. 2020. Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks. International Journal of Computer Vision, 128:891 – 908. find
  16. Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. 2018. Sign language production using neural machine translation and generative adversarial networks. In Proceedings of the 29th British Machine Vision Conference (BMVC 2018). British Machine Vision Association. find
  17. Valerie Sutton and Adam Frost. 2008. SignWriting: sign languages are written languages. Center for Sutton Movement Writing. find
  18. Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30. find
  19. Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. 2024. G2P-DDM: Generating sign pose sequence from gloss sequence with discrete diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6234–6242. find
  20. J Zhang, Y Zhang, X Cun, S Huang, Y Zhang, H Zhao, H Lu, and X Shen. 2023. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. arXiv preprint arXiv:2301.06052. — [arXiv:2301.06052](https://arxiv.org/abs/2301.06052) find
  21. Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. 2024. MotionGPT: Finetuned LLMs are General-Purpose Motion Generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368–7376. find
  22. Inge Zwitserlood, Margriet Verlinden, Johan Ros, and Sanny van der Schoot. 2005. Synthetic signing for the deaf: eSIGN. 8. find
  23. Language Resource References Thomas Hanke. 2009. DGS corpus project - Development of a corpus based electronic dictionary German Sign Language / German. find
  24. Silke Matthes, Thomas Hanke, Anja Regen, Jakob Storz, Satu Worseck, Eleni Efthimiou, Athanasia-Lida Dimou, Annelies Braffort, John R. W. Glauert, and Éva Sáfár. 2012. Dicta-Sign – Building a Multilingual Sign Language Corpus. find
  25. Joanna Łacheta and Paweł Rutkowski. 2014. A Corpus-based Dictionary of Polish Sign Language ( PJM ). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/maximo-2026-generating-sign-language-poses-hamnosys/