Elamur Research⌕
☰
home / papers / 2026 / MSCode: Advancing Human Motion-Language Understanding via Modality-Shared Codebook

MSCode: Advancing Human Motion-Language Understanding via Modality-Shared Codebook

Paper · 2026 · Haoyu Shi and Huaiwen Zhang

Summary

MSCode tackles a core difficulty in human motion-language understanding. Motion sequences carry detailed spatiotemporal information, while their textual descriptions are short and abstract. This gap in semantic granularity weakens cross-modal alignment in both global contrastive approaches (e.g., TMR) and vocabulary-space approaches (e.g., ToHL).

The authors place both modalities in a common space defined by a learnable, modality-shared codebook. Each motion and each sentence is expressed as a weighted mix of the same codewords. Two mechanisms push a matched pair to select the same small subset of codewords: sparse activation and a distribution-consistency objective based on symmetric KL divergence. Two further components support this:

  • a locality-aware Gaussian encoder, which restricts each pose to attend to nearby frames so that pose embeddings do not collapse toward similarity;
  • a hard-negative guided contrastive loss, which sharpens discrimination between pairs.

According to the authors, MSCode outperforms prior state-of-the-art methods on text-motion retrieval, text-to-motion grounding and motion captioning. It is also more efficient than lexical-space matching. The work matters because a compact shared semantic basis is a practical alternative to very high-dimensional vocabulary spaces for fine-grained motion-text alignment.

Key points

  • Text is encoded with a pre-trained DistilBERT plus a projection head. Poses, which include joint positions, rotations and foot contact, are encoded with the proposed Gaussian encoder.
  • ToHL works in a lexical space with 30,522 dimensions (the BERT vocabulary size). MSCode's codebook is far more compact, and the ablations favour a codebook size of 1024 for retrieval.
  • In zero-shot grounding, MSCode keeps a high IoU on queries containing several actions, whereas CAR degrades on such queries. In qualitative retrieval examples, it also captures complex multi-action queries better than CAR and ToHL.
  • Limitations noted by the authors:
    • A fixed codebook size involves a trade-off: grounding benefits from a larger codebook than retrieval does.
    • Sparse activation and the consistency loss add training overhead as the codebook grows.
  • Published in Findings of ACL 2026 by authors from Inner Mongolia University.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksControl and Systems Engineering · 45 works

Cite

Cite as Haoyu Shi and Huaiwen Zhang. 2026. MSCode: Advancing Human Motion-Language Understanding via Modality-Shared Codebook. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1901

@inproceedings{shi2026mscode,
  title = {{MSCode: Advancing Human Motion-Language Understanding via Modality-Shared Codebook}},
  author = {Haoyu Shi and Huaiwen Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1901},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1901}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael U. Gutmann and tevet2022motionclipAapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. 13. find
  2. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. Beit: Bert pre-training of image transformers. Preprint, arXiv:2106.08254. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
  3. Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. Motiongpt: Human motion as a foreign language. Preprint, arXiv:2306.14795. — [arXiv:2306.14795](https://arxiv.org/abs/2306.14795) find
  4. Chuhao Jin, Rui Zhang, Qingzhe Gao, Haoyu Shi, Dayu Wu, Yichen Jiang, Yihan Wu, and Ruihua Song. 2026. Sentiavatar: Towards expressive and interactive digital humans. arXiv preprint arXiv:2604.02908. — [arXiv:2604.02908](https://arxiv.org/abs/2604.02908) find
  5. Mathis Petrovich, Michael J. Black, and Gül Varol. 2022. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV). find
  6. Sai Shashank Kalakonda, Shubh Maheshwari, and Ravi Kiran Sarvadevabhatla. 2022. Action-gpt: Leveraging large-scale language models for improved and generalized action generation. 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 31–36. find
  7. Mathis Petrovich, Michael J. Black, and Gül Varol. 2023. TMR: Text-to-motion retrieval using contrastive 3D human motion synthesis. In International Conference on Computer Vision (ICCV). find
  8. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  9. Solomon Kullback and Richard A Leibler. 1951. On information and regression. The Annals of Mathematical Statistics, 22(1):79–86. find
  10. Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. 2024. Intergen: Diffusion-based multi-human motion generation under complex interactions. International Journal of Computer Vision, pages 1– 21. find
  11. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  12. Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems. find
  13. Guangtao Lyu, Chenghao Xu, Jiexi Yan, Muli Yang, and Cheng Deng. 2025. Towards unified human motion-language understanding via sparse interpretable characterization. In The Thirteenth International Conference on Learning Representations. find
  14. Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019. AMASS: archive of motion capture as surface shapes. In ICCV 2019, Seoul, Korea (South), October 27 -November 2, 2019, pages 5441–5450. IEEE. find
  15. André F. T. Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. Preprint, arXiv:1602.02068. find
  16. Nicola Messina, Jan Sedmidubský, Fabrizio Falchi, and Tomás Rebok. 2023. Text-to-motion retrieval: Towards joint understanding of human motion data and natural language. In ACM SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 2420–2425. ACM. find
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics.
  18. Matthias Plappert, Christian Mandery, and Tamim Asfour. 2016. The KIT motion-language dataset. Big Data, 4(4):236–252. find
  19. Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black. 2021. BABEL: Bodies, action and behavior with english labels. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 722–731. find
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  21. Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. Preprint, arXiv:2010.04592. — [arXiv:2010.04592](https://arxiv.org/abs/2010.04592) find
  22. Haoyu Shi and Huaiwen Zhang. 2024. Modal-enhanced semantic modeling for fine-grained 3d human motion retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 10114– 10123. find
  23. Haoyu Shi and Huaiwen Zhang. 2025. Sequence-event semantic consistent learning for text-to-motion retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8740–8749. find
  24. Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In CVPR. find
  25. Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, and Daniel Cohen-Or. 2022. Motionclip: Exposing human motion generation to clip space. Preprint, arXiv:2203.08063. — [arXiv:2203.08063](https://arxiv.org/abs/2203.08063) find
  26. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2018. Neural discrete representation learning. Preprint, arXiv:1711.00937. find
  27. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566– 4575.
  28. Sheng Yan, Mengyuan Liu, Yong Wang, Yang Liu, Chen Chen, and Hong Liu. 2024. Mlp: Motion label prior for temporal sentence localization in untrimmed 3d human motions. Preprint, arXiv:2404.13657. — [arXiv:2404.13657](https://arxiv.org/abs/2404.13657) find
  29. Sheng Yan, Yang Liu, Haoqiang Wang, Xin Du, Mengyuan Liu, and Hong Liu. 2023. Cross-modal retrieval for motion and text via droptriple loss. In ACM Multimedia Asia 2023, MMAsia ’23. ACM. find
  30. Yang Yang, Liyuan Cao, Haoyu Shi, and Huaiwen Zhang. 2024a. Multi-instance multi-label learning for text-motion retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5829–5837. find
  31. Yang Yang, Haoyu Shi, and Huaiwen Zhang. 2024b. Hierarchical semantics alignment for 3d human motion retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1083–1092. find
  32. Yang Yang, Haoyu Shi, and Huaiwen Zhang. 2024c. Hierarchical semantics alignment for 3d human motion retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 1083–1092. ACM. find
  33. Qing Yu, Mikihiro Tanaka, and Kent Fujiwara. 2024. Exploring vision transformers for 3d human motion-language models with motion patches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  34. Jiahang Zhang, Lilang Lin, Shuai Yang, and Jiaying Liu. Sgar: Structural generative augmentation for 3d human motion retrieval. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  35. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-mscode-advancing-human-motion-language/