Elamur Research⌕
☰
home / papers / 2026 / Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production

Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production

Paper · 2026 · Maoxiao Ye, Xinfeng Ye and Sathiamoorthy Manoharan

Summary

Sign Language Production (SLP) converts text into sequences of signer poses. Autoregressive generators work frame by frame, but their errors compound over time. Diffusion-based generators produce better poses, but they denoise whole sequences, so the first output arrives late. The authors present HybridSign, which applies a causal mask to the self-attention of a flow-matching diffusion denoiser. Each frame is therefore refined by diffusion but still emitted in temporal order.

Three further design choices support this:

  • Keypoints are split into face, body and hands, each handled by a dedicated expert before the parts are merged.
  • Joint-level confidence scores from 2D pose extraction steer attention away from unreliable joints.
  • A self-forcing scheme conditions training on the model's own earlier outputs, with a Soft-DTW loss added for sequence-level supervision.

On PHOENIX14T and How2Sign, the system offers the best balance of output quality and speed among the compared baselines. This matters for interactive applications such as sign interpreters, where the delay before the first frame appears is critical.

Key points

  • On the How2Sign test split, HybridSign reaches BLEU-1/4 of 30.12/6.48 and DTW of 3.89. Under a 60-frame evaluation protocol, it achieves a time-to-first-frame of 5.90 s and 10.17 FPS throughput.
  • In ablations, a pure DiT diffusion model scored the highest BLEU but had poor latency and weaker temporal consistency. A teacher-forced Transformer decoder started quickly but lost quality.
  • Confidence-aware causal attention beat both an RNN and standard causal attention on quality. The RNN had the lowest first-frame latency but lower throughput.
  • Three experts (face, body, both hands) outperformed both one expert and four experts. Separating the two hands weakens modeling of coordinated bimanual motion.
  • Stated limitations include reliance on 2D poses (depth ambiguity during occlusion or hand-face contact), limited datasets, and incomplete modeling of non-manual signals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHuman-Computer Interaction · 160 works

Cite

Cite as Maoxiao Ye, Xinfeng Ye and Sathiamoorthy Manoharan. 2026. Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.31

@inproceedings{ye2026hybrid,
  title = {{Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production}},
  author = {Maoxiao Ye, Xinfeng Ye and Sathiamoorthy Manoharan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.31},
  url = {https://doi.org/10.18653/v1/2026.acl-long.31}
}

References

39 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  2. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020a. Multi-channel transformers for multi-articulatory sign language translation. Preprint, arXiv:2009.00299. — [arXiv:2009.00299](https://arxiv.org/abs/2009.00299) find
  3. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020b. Sign language transformers: Joint end-to-end sign language recognition and translation. Preprint, arXiv:2003.13830. — [arXiv:2003.13830](https://arxiv.org/abs/2003.13830) find
  4. Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  5. Marco Cuturi and Mathieu Blondel. 2018. Soft-dtw: a differentiable loss function for time-series. Preprint, arXiv:1703.01541. — [arXiv:1703.01541](https://arxiv.org/abs/1703.01541) find
  6. Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. In Conference on Computer Vision and Pattern Recognition (CVPR). find
  7. Sen Fang, Chunyu Sui, Yanghao Zhou, Xuedong Zhang, Hongbin Zhong, Yapeng Tian, and Chen Chen. 2025. Signdiff: Diffusion model for american sign language production. Preprint, arXiv:2308.16082. — [arXiv:2308.16082](https://arxiv.org/abs/2308.16082) find
  8. Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. Llms are good sign language translators. Preprint, arXiv:2404.00925. — [arXiv:2404.00925](https://arxiv.org/abs/2404.00925) find
  9. Alex Graves. 2012. Sequence transduction with recurrent neural networks. Preprint, arXiv:1211.3711.
  10. Dan Guo, Shengeng Tang, and Meng Wang. 2019. Connectionist temporal modeling of video and language: a joint model for translation and sign labeling. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 751–757. find
  11. Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. 2017. Online early-late fusion based on adaptive hmm for sign language recognition. ACM Transactions on Multimedia Computing, Communications, and Applications, 14(1):1–18. find
  12. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Preprint, arXiv:2006.11239. find
  13. Hezhen Hu, Junfu Pu, Wengang Zhou, and Houqiang Li. 2023a. Collaborative multilingual continuous sign language recognition: A unified framework. IEEE Transactions on Multimedia, 25:7559–7570. find
  14. Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. 2023b. Signbert+: Hand-model-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11221–11239. find
  15. Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian. 2021. Towards fast and high-quality sign language production. Proceedings of the 29th ACM International Conference on Multimedia. find
  16. Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. 2025. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009.
  17. Oscar Koller. 2020. Quantitative survey of the state of the art in sign language recognition. Preprint, arXiv:2008.09918. — [arXiv:2008.09918](https://arxiv.org/abs/2008.09918) find
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow matching for generative modeling. Preprint, arXiv:2210.02747.
  20. Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. 2024. Flow matching guide and code. Preprint, arXiv:2412.06264. — [arXiv:2412.06264](https://arxiv.org/abs/2412.06264) find
  21. Shikun Liu, Edward Johns, and Andrew J. Davison. 2019. End-to-end multi-task learning with attention. Preprint, arXiv:1803.10704. find
  22. Xingchao Liu, Chengyue Gong, and Qiang Liu. 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. Preprint, arXiv:2209.03003.
  23. Seshadri Mazumder, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C. V. Jawahar. 2021. Translating sign language videos to talking faces. In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP). find
  24. John McDonald, Rosalee J. Wolfe, Jerry Schnepp, Julie Hochgesang, Diana Gorman Jamrozik, Marie Stumbo, Larwan Berke, Melissa Bialek, and Farah Thomas. 2016. An automated technique for real-time production of lifelike animations of american sign language. Universal Access in the Information Society,… find
  25. Alex Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. Preprint, arXiv:2102.09672. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  27. Shengeng Tang, Feng Xue, Jingjing Wu, Shuo Wang, and Richang Hong. 2025. Gloss-driven conditional diffusion models for sign language production. ACM Trans. Multimedia Comput. Commun. Appl., 21(4). find
  28. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. Preprint, arXiv:2212.09748.
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Preprint, arXiv:1706.03762.
  30. Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020a. Adversarial training for multi-channel sign language production. Preprint, arXiv:2008.12405. — [arXiv:2008.12405](https://arxiv.org/abs/2008.12405) find
  31. Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020b. Progressive transformers for end-to-end sign language production. Preprint, arXiv:2004.14874. — [arXiv:2004.14874](https://arxiv.org/abs/2004.14874) find
  32. Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. 2024. G2p-ddm: Generating sign pose sequence from gloss sequence with discrete diffusion model. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6). find
  33. Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li. 2023. Best: Bert pre-training for sign language recognition with coupling tokenization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3597–3605. find
  34. Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021. Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks. Preprint, arXiv:2103.06982. — [arXiv:2103.06982](https://arxiv.org/abs/2103.06982) find
  35. A Jérémie Segouat. 2009. A study of sign language coarticulation. ACM Sigaccess Accessibility and Computing, 93. find
  36. This appendix provides supplementary details on data preprocessing, training configuration, evaluation protocols, and additional qualitative results. find
  37. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. Preprint, arXiv:2010.02502.
  38. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-based generative modeling through stochastic differential equations. Preprint, arXiv:2011.13456.
  39. Stephanie Stoll, Necati Camgoz, Simon Hadfield, and Richard Bowden. 2020. Text2sign: Towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-hybrid-autoregressive-diffusion-model-real/