Elamur Research⌕
☰
home / papers / 2025 / Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

Summary

Vision-and-Language Navigation (VLN) research is almost entirely English. This paper asks whether language models can plan indoor routes from Arabic instructions as well as from English ones, and whether an Arabic-centric model helps. The authors machine-translated the R2R dataset into Arabic with the Groq API. The translation covers the instructions, the textual scene observations and the object lists, which gives a bilingual VLN benchmark. They then ran four models inside the zero-shot NavGPT framework: GPT-4o mini, Llama 3 8B, Phi-3 medium 14B and Jais 30B. NavGPT does not process images directly. It describes the visual scene in text and asks the model to reason and pick the next viewpoint step by step.

GPT-4o mini planned well in both languages and beat Llama 3 8B roughly fivefold. Phi-3 was weak in Arabic, which the authors attribute to it not being multilingual. Jais scored zero success rate in both languages. The main takeaway is that general reasoning strength mattered more than the instruction language or Arabic specialisation. This matters for deploying robots in Arabic-speaking regions.

Key points

  • The authors describe this as the first VLN study that integrates Arabic. The models are GPT-4o mini, Llama 3 8B and Phi-3 medium 14B, plus the Arabic-centric LLM Jais 30B.
  • The benchmark is R2R (on Matterport3D environments), extended with Arabic machine translations from the Groq API. The authors note that RxR covers several languages but not Arabic.
  • NavGPT works zero-shot. At each step it receives three kinds of text: descriptions of the current observation, a one-sentence summary of the navigation history, and the list of navigable viewpoints.
  • Results:
    • GPT-4o mini was about five times better than Llama 3 8B.
    • Jais reached 0 SR in both English and Arabic.
    • Phi-3 struggled in Arabic.
  • Failure types:
    • Parsing errors: malformed output, missing actions, or invented viewpoints that do not exist.
    • Jais-specific failures: it could not decompose goals or keep its steps consistent.
    • Proposed fixes: human translation, the Arabic models SILMA and ALLaM, and task-specific fine-tuning.

Interesting ideas

  • I1. The authors translate every model input (instructions, scene observations, object lists) and also let the model reason in Arabic, so the whole decision loop runs in one language rather than mixing translated commands with English context. This gives a cleaner test of language effects. — For Mova (learn): Mova's evaluation could feed the same task through a paraphrased or otherwise surface-altered version of the full input as a control, checking that the frames and the answer stay stable when only wording changes.
  • I2. Failures are split into format/parsing breakdowns versus genuine planning breakdowns, with hallucinated non-existent viewpoints counted as their own kind of error. This keeps interface bugs from being mistaken for reasoning weakness. — For Mova (answer): Mova's readers could reject any answer that names an entity or location absent from the built world of the text, and log such cases separately from wrong-but-grounded answers.
  • I3. Perception is swapped for textual scene descriptions, so the planner reasons only over language. This decouples planning quality from vision and narrows the sim-to-real gap. — For Mova (world): route instructions over a text-described space (rooms, objects, connections) would make a good test set for Mova's tracking of locations, states and event order.
  • I4. The authors admit that squeezing navigation history into one summary sentence drops earlier details, which hurts later steps. This is an argument for keeping history structured instead of compressed into prose. — For Mova (world): Mova should keep an explicit, queryable log of entity states and past events rather than a running textual summary, so that late questions can still reach early facts.
  • I5. An Arabic-specialised model lost to a general strong reasoner even on Arabic input. This suggests that per-language specialisation does not replace sound reasoning machinery. — For Mova (frames): when the frame layer is extended to new constructions or registers, reusing the general role-assignment logic is likely to pay off more than building separate special-case modules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Multilingual & cross-lingual NLP · Semitic languages

Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates33 works · 3 016 citations🔬Inception Institute of Artificial IntelligenceAbu Dhabi · United Arab Emirates4 works · 269 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.05478 · arXiv 2501.05478

@misc{mansour2025language,
  title = {{Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models}},
  author = {Mansour, Malak and Aly, Ahmed and Tharwat, Bahey and Hashmi, Sarim and An, Dong and Reid, Ian},
  year = {2025},
  doi = {10.48550/arxiv.2501.05478},
  eprint = {2501.05478},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.05478}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, M.; et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219. ↗ our card
  2. Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; Sünderhauf, N.; Reid, I.; Gould, S.; and van den Hengel, A. 2018. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. arXiv:1711.07280. ↗ our card
  3. Bari, M. S.; Alnumay, Y.; Alzahrani, N. A.; Alotaibi, N. M.; Alyahya, H. A.; AlRashed, S.; Mirza, F. A.; Alsubaie, S. Z.; Alahmed, H. A.; Alabduljabbar, G.; Alkhathran, R.; Almushayqih, Y.; Alnajim, R.; Alsubaihi, S.; Mansour, M. A.; Alrubaian, M.; Alammari, A.; Alawami, Z.; Al-Thubaity, A.; Abdelali, A.; Kuriakose, J.; Abujabal, A.; Al-Twairesh, N.; Alowisheq, A.; and Khan, H. 2024. ALLaM: Large Language Models for Arabic and English. arXiv:2407.15390. ↗ our card
  4. Chang, A.; Dai, A.; Funkhouser, T.; Halber, M.; Niessner, M.; Savva, M.; Song, S.; Zeng, A.; and Zhang, Y. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. International Conference on 3D Vision (3DV). find
  5. Dubey, A. 2024. The Llama 3 Herd of Models. arXiv:2407.21783. ↗
  6. Ghallab, M.; Nau, D.; and Traverso, P. 2016. Automated Planning and Acting. Cambridge University Press. find
  7. Gu, J.; Stefani, E.; Wu, Q.; Thomason, J.; and Wang, X. 2022. Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. find
  8. Jiang, Y.; Zhang, S.; Khandelwal, P.; and Stone, P. 2019. Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systems. arXiv:1804.08229. ↗
  9. Khalati, M.; Ali, T.; and Al-Romany, T. 2024. Artificial Intelligence Development and Challenges (Arabic Language as a Model). find
  10. Koto, F.; Li, H.; Shatnawi, S.; Doughman, J.; Sadallah, A. B.; Alraeesi, A.; Almubarak, K.; Alyafeai, Z.; Sengupta, N.; Shehata, S.; Habash, N.; Nakov, P.; and Baldwin, T. 2024. ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic. arXiv:2402.12840. ↗ our card
  11. Ku, A.; Anderson, P.; Patel, R.; Ie, E.; and Baldridge, J. 2020. Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding. arXiv:2010.07954. ↗
  12. Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. arXiv:1908.02265. ↗ our card
  13. Lu, Z.; Li, X.; Cai, D.; Yi, R.; Liu, F.; Zhang, X.; Lane, N. D.; and Xu, M. 2024. Small Language Models: Survey, Measurements, and Insights. arXiv:2409.15790. ↗
  14. OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774. ↗ our card
  15. Rahmanzadehgervi, P.; Bolton, L.; Taesiri, M. R.; and Nguyen, A. T. 2024. Vision language models are blind. arXiv:2407.06581. ↗
  16. Sengupta, N.; Sahu, S.; Jia, B.; Katipomu, S.; Li, H.; Koto, F.; Afzal, O.; Kamboj, S.; Pandit, O.; Pal, R.; Pradhan, L.; Muhammad Mujahid, Z.; Baali, M.; Aji, A.; Liu, Z.; Hock, A.; Feldman, A.; Lee, J.; Jackson, A.; and Xing, E. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. find our card
  17. Tan, H.; Yu, L.; and Bansal, M. 2019. Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout. arXiv:1904.04195. ↗
  18. Vargas-Munoz, J. E.; Srivastava, S.; Tuia, D.; and Falcao, A. X. 2021. OpenStreetMap: Challenges and Opportunities in Machine Learning and Remote Sensing. IEEE Geoscience and Remote Sensing Magazine, 9(1): 184–199. find
  19. Vidhyashree, A. 2024. GPT-4O Mini: The New Lightweight Version of GPT-4. [Accessed: Nov. 19, 2024]. find
  20. Zhang, Y.; Ma, Z.; Li, J.; Qiao, Y.; Wang, Z.; Chai, J.; Wu, Q.; Bansal, M.; and Kordjamshidi, P. 2024. Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models. arXiv:2407.07035. ↗
  21. Zhou, G.; Hong, Y.; Wang, Z.; Wang, X. E.; and Wu, Q. 2024. NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models. arXiv:2407.12366. ↗
  22. Zhou, G.; Hong, Y.; and Wu, Q. 2023. NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models. arXiv:2305.16986. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mansour-2025-language-planning-robotic-navigation-multilingual/