Elamur Research⌕
☰
home / papers / 2026 / Limited Linguistic Diversity in Embodied AI Datasets

Limited Linguistic Diversity in Embodied AI Datasets

Paper · 2026 · Selma Liliane Wanna, Agnes Luhtaru, Jonathan Salfity et al.

Summary

Language is a core input for Vision-Language-Action (VLA) models, but the instructions in their training data are rarely documented. This makes it hard to explain known weaknesses such as sensitivity to paraphrases. The authors audit instruction text in several popular datasets from the Open X-Embodiment (OXE) collection, including RT-1, BridgeData and Language Table. They compare these with other robotics datasets and with instruction-tuning corpora used for LLMs. The audit has three parts:

  • Lexical: duplication and word-level variety.
  • Semantic: embedding-based measures plus verb–object–adverbial combinations.
  • Structural: POS patterns and phenomena such as negation and conditionals.

The picture is consistent across datasets. Commands are heavily repeated and template-like, short, and built from a small vocabulary. Richer constructions are almost absent. The authors frame this as descriptive documentation rather than a causal claim about model performance. From it they derive practical advice: targeted paraphrase and synonym augmentation guided by the metrics, and data-collection interfaces that push annotators toward more varied phrasing. Interactive collection in the style of SCOUT's Wizard-of-Oz setup already yields more diverse language.

Key points

  • Under 2% of instructions in the studied VLA datasets are unique, and lexical diversity falls below the comparison corpora.
  • Negation appears in less than 1% of commands, and conditionals are similarly scarce.
  • Lexical metrics: ROUGE-L and gzip compression ratio (CR) are reported in the main text; BLEU, Jaccard similarity and Levenshtein distance are in the appendix.
  • Semantic metrics: pairwise BERTScore, plus the number of PCA components needed for 95% of variance. These use USE, SBERT, CLIP and SONAR embeddings.
  • Structural findings: the single most frequent POS pattern makes up 11% of RT-1 and 24% of TacoPlay. Commands typically run 3–15 words, and RT-1 shows strong verb–noun co-occurrence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations🎓University of Maryland, Baltimore CountyBaltimore · United States14 works · 915 citations🎓University of TartuTartu · Estonia12 works · 2,874 citations🔬Los Alamos National LaboratoryLos Alamos · United States3 works · 61,941 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Selma Liliane Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek and Mitch Pryor. 2026. Limited Linguistic Diversity in Embodied AI Datasets. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.507

@inproceedings{wanna2026limited,
  title = {{Limited Linguistic Diversity in Embodied AI Datasets}},
  author = {Selma Liliane Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek and Mitch Pryor},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.507},
  url = {https://doi.org/10.18653/v1/2026.acl-long.507}
}

References

66 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arshiya Aggarwal, Jiao Sun, and Nanyun Peng. 2022. Towards robust NLG bias evaluation with syntactically-diverse prompts. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6022–6032, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  2. AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, and 33 others. 2025. Agibot world colosseo: A large-scale manipulatio… — [arXiv:2503.06669](https://arxiv.org/abs/2503.06669) find
  3. Emily M. Bender. 2019. The benderrule: On naming the languages we study and why it matters. The Gradient. find
  4. Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, and 16 others. 2024. Paligemma: A versatile 3b vlm for transfer. Preprint, arXiv:2407.07726. find
  5. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, and 5 others. 2024. π 0 :… find
  6. Anthony Brohan and 1 others. 2023. Rt-1: Robotics transformer for real-world control at scale. Preprint, arXiv:2212.06817. find
  7. Daniel Cer and 1 others. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  8. William Chen, Suneel Belkhale, Suvir Mirchandani, Oier Mees, Danny Driess, Karl Pertsch, and Sergey Levine. 2025. Training strategies for efficient embodied reasoning. Preprint, arXiv:2505.08243. — [arXiv:2505.08243](https://arxiv.org/abs/2505.08243) find
  9. Embodiment Collaboration and 1 others. 2024. Open x-embodiment: Robotic learning datasets and rt-x models. Preprint, arXiv:2310.08864. find
  10. Shivin Dass, Alaa Khaddaj, Logan Engstrom, Aleksander Madry, Andrew Ilyas, and Roberto Martín-Martín. 2025. Datamil: Selecting data for robot imitation learning with datamodels. Preprint, arXiv:2505.09603. — [arXiv:2505.09603](https://arxiv.org/abs/2505.09603) find
  11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  12. Mathieu Dehouck and Carlos Gómez-Rodríguez. 2020. Data augmentation via subtree swapping for dependency parsing of low-resource languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3818–3830, Barcelona, Spain (Online). International Committee on Computat… find
  13. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  14. Danny Driess, Jost Tobias Springenberg, brian ichter, LILI YU, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, and Sergey Levine. 2025. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better. In The Thirty-ninth Annual Conf… find
  15. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Paul-Ambroise Duquenne and 1 others. 2023. SONAR: sentence-level multimodal and language-agnostic representations. arXiv preprint. find
  16. Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. 2022. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics: Science and Systems. find
  17. Mingyu Fan, Nannan Gu, Hong Qiao, and Bo Zhang. 2010. Intrinsic dimension estimation of data by principal component analysis. Preprint, arXiv:1002.2050. — [arXiv:1002.2050](https://arxiv.org/abs/1002.2050) find
  18. Jensen Gao, Suneel Belkhale, Sudeep Dasari, Ashwin Balakrishna, Dhruv Shah, and Dorsa Sadigh. 2025. A taxonomy for evaluating generalist robot policies. Preprint, arXiv:2503.01238. — [arXiv:2503.01238](https://arxiv.org/abs/2503.01238) find
  19. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong X… find
  20. Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I. Christensen, Hao Su, and Xuanlin Li. 2025. Enhancing generalization in vision-language-action models by preserving pretrained representations. Preprint, arXiv:2509.11417. — [arXiv:2509.11417](https://arxiv.org/abs/2509.11417) find
  21. Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2024. The curious decline of linguistic diversity: Training language models on synthetic text. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3589–3604, Mexico City, Mexico. Association for Computatio… find
  22. Joey Hejna, Chethan Anand Bhateja, Yichen Jiang, Karl Pertsch, and Dorsa Sadigh. 2025. Remix: Optimizing data mixtures for large scale imitation learning. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 145–164. PMLR. find
  23. Matthew Honnibal and 1 others. 2020. spacy: Industrial-strength natural language processing in python. find
  24. Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723, Dublin, Ireland. Association for Com… find
  25. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, and 17 others. 2… find
  26. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. PMLR.
  27. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2024. Openvla: An open-source… find
  28. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. Openvla: An open-source vision-… find
  29. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alex… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  30. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  31. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  32. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023a. Libero: Benchmarking knowledge transfer for lifelong robot learning. Preprint, arXiv:2306.03310. find
  33. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.
  34. Stephanie M. Lukin and 1 others. 2024. SCOUT: A situated and multi-modal human-robot dialogue corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14445–14458, Torino, Italia. ELRA and ICCL. find
  35. Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. 2023. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, pages 1–8. find
  36. Yash Mahajan, Naman Bansal, Eduardo Blanco, and Santu Karmaker. 2024. ALIGN-SIM: A task-free test bed for evaluating and interpreting sentence embeddings through semantic similarity alignment. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7393–7428, Miami, Florida,… find
  37. Alessandro Moschitti. 2006. Making tree kernels practical for natural language learning. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 113–120, Trento, Italy. Association for Computational Linguistics. find
  38. The pandas development team. 2020. dev/pandas: Pandas. pandas-Kishore Papineni and 1 others. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, US…
  39. Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. 2025. Fast: Efficient action tokenization for vision-language-action models. Preprint, arXiv:2501.09747.
  40. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. arXiv preprint.
  41. Nils Reimers and 1 others. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982– 3992, Hong…
  42. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. find
  43. Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, and Ani Nenkova. 2025. Standardizing the measurement of text diversity: A tool and a comparative analysis of scores. Preprint, arXiv:2403.00553. — [arXiv:2403.00553](https://arxiv.org/abs/2403.00553) find
  44. Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for nlp. Preprint, arXiv:2101.00411. find
  45. Mohit Shridhar and 1 others. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  46. Katherine Stasaski and Marti Hearst. 2022. Semantic diversity in dialogue with natural language inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 85–98, Seattle, United States. Associati… find
  47. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  48. Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326–346, Online. Association for Computational Linguistics.
  49. Fumiya Uchiyama, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, and Yutaka Matsuo. 2024. Which programming language and what features at pre-training stage affect downstream logical inference performance? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro… find
  50. Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. 2024. No ”zero-shot” without exponential data: Pretraining concept frequency determines multimodal model performance. In The Thirty-eighth Annual Conference on Neural Information… find
  51. Erick Rosete-Beas and 1 others. 2022. Latent plans for task agnostic offline reinforcement learning. find
  52. Michel Verleysen and John A. Lee. 2013. Nonlinear dimensionality reduction for visualization. In Neural Information Processing, pages 617–622, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  53. Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. 2020. The pitfalls of simplicity bias in neural networks. In Homer Walke and 1 others. 2023. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL). find
  54. Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, and Lei Ma. 2024. Ladev: A language-driven testing and evaluation platform for vision-language-action models in robotic manipulation. Preprint, arXiv:2410.05191. — [arXiv:2410.05191](https://arxiv.org/abs/2410.05191) find
  55. Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Heng Tao Shen, and Jingkuan Song. 2025. Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation. In 9th Annual Conference on Robot Learning. find
  56. Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. 2024. Robotic control via embodied chain-of-thought reasoning. In 8th Annual Conference on Robot Learning. find
  57. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations. find
  58. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  59. Yan Zhang, Zhaopeng Feng, Zhiyang Teng, Zuozhu Liu, and Haizhou Li. 2023. How well do text embedding models understand syntax? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9717–9728, Singapore. Association for Computational Linguistics. find
  60. Yubo Zhang and 1 others. 2020. Diagnosing the environment bias in vision-and-language navigation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 890–897. International Joint Conferences on Artificial Intelligence Organization. Main track… find
  61. Yuhui Zhang, Yuchang Su, Yiming Liu, and Serena Yeung-Levy. 2025. NegVQA: Can vision language models understand negation? In Findings of the Association for Computational Linguistics: ACL 2025, pages 3707–3716, Vienna, Austria. Association for Computational Linguistics. find
  62. Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024. Why are visually-grounded language models bad at image classification? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  63. Dora Zhao, Jerone Andrews, Orestis Papakyriakopoulos, and Alice Xiang. 2024. Position: Measure dataset diversity, don’t just claim it. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 60644–60673. PMLR. find
  64. Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, and Feifei Feng. 2025. find
  65. Chatvla: Unified multimodal understanding and robot control with vision-language-action model. Preprint, arXiv:2502.14420. find
  66. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1097–1100, New York, NY, USA. Assoc…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wanna-2026-limited-linguistic-diversity-embodied-ai/