Elamur Research⌕
☰
home / papers / 2024 / Airavata: Introducing Hindi Instruction-tuned LLM

Airavata: Introducing Hindi Instruction-tuned LLM

Paper · 2024 · Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain et al.

Summary

Open instruction-following LLMs are built mostly for English. Hindi and other Indian languages get weaker tokenization, less training data and lower task quality, even in commercial systems. The AI4Bharat team presents Airavata, a Hindi assistant model made by instruction-tuning OpenHathi, a Llama 2 derivative from Sarvam AI. They deliberately avoid outputs distilled from proprietary models such as GPT-4. Instead, they machine-translate human-written, permissively licensed English instruction sets into Hindi with IndicTrans2 and keep only translations that pass a back-translation quality check. They also add two new native resources, wikiHow and Anudesh. Training uses LoRA rather than full fine-tuning, and the final model blends two epoch checkpoints by weight interpolation. The authors also assemble a Hindi evaluation suite and use it, together with human judgments, to compare LLMs. Instruction tuning clearly helps Hindi NLU tasks, but generation (NLG) still lags, and the model can hallucinate facts or ignore instructions. The released data, code and benchmarks give IndicLLM research a reusable starting point.

Key points

  • Training data: About 404k English+Hindi examples were drawn from FLAN-v2, Anthropic-HHH, Dolly, OpenAssistant, LymSys-Chat, wikiHow, Anudesh and 50k BPCC-Human translation pairs. Filtering with chrF++ ≥ 50 on back-translations left about 385k examples, published as ai4bharat/indic-instruct-data-v0.1.
  • New native datasets:
    • wikiHow: roughly 20k English and 6k Hindi how-to articles, framed as completion tasks, licensed CC-0.
    • Anudesh: crowd-sourced prompts, some code-mixed or transliterated, paired with Llama 2 70B responses, licensed CC-BY-4.0.
  • LoRA settings: rank 16, alpha 32, dropout 0.05, 4 epochs, learning rate 5e-4, batch size 128, bfloat16. Loss was computed on output tokens only, using IndicInstruct, a customized fork of OpenInstruct.
  • Full fine-tuning vs. LoRA: In an ablation on the FLAN-v2 subset, full fine-tuning improved IndicCOPA and IndicXParaphrase but hurt English BoolQ and MMLU. LoRA matched or beat the base model in both languages.
  • Final checkpoint: The released weights are 0.6 × epoch-3 plus 0.4 × epoch-4. Epoch 3 was stronger on NLU and epoch 4 on NLG, so the mix balances the two.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🏢IBMArmonk · United States139 works · 70,691 citations🎓Indian Institute of Technology MadrasChennai · India16 works · 2,499 citations🔬National Institute of Information and Communications TechnologyTokyo · Japan23 works · 2,090 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

Also on the byline: Indian Institute of Information Technology, Design and Manufacturing, Kancheepuram.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy and Anoop Kunchukuttan. 2024. Airavata: Introducing Hindi Instruction-tuned LLM. arXiv:2401.15006.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.15006 · arXiv 2401.15006

@misc{gala2024airavata,
  title = {{Airavata: Introducing Hindi Instruction-tuned LLM}},
  author = {Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy and Anoop Kunchukuttan},
  year = {2024},
  doi = {10.48550/arxiv.2401.15006},
  eprint = {2401.15006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.15006}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report. 2023. URL https://api.semanticscholar.org/CorpusID:257532815. ↗
  2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative AI. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of t…
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 2204.05862, 2022. find
  4. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions, October 2023. URL https://cdn.openai.com/papers/… ↗
  5. Hyung Won Chung, Le Hou, Shayne Longpre, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv: 2210.11416, 2022. find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapte…
  7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457, 2018.
  8. Together Computer. Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  9. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/ 12/dolly-first-open-commercially-viable-… ↗
  10. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semar… find
  11. Mithun Das, Punyajoy Saha, Binny Mathew, and Animesh Mukherjee. HateCheckHIn: Evaluating Hindi hate speech detection models. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hél… find
  12. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
  13. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreyansh Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. Annual Meeting of the Association for Computational Ling…
  14. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  15. Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan. Indictrans2: Towards high-quality and accessible machine translatio… — IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
  16. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. 2021. find
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KB jmI3GmQ. ↗
  18. Saghar Hosseini, Hamid Palangi, and Ahmed Hassan Awadallah. An empirical study of metrics to measure representational harms in pre-trained language models. arXiv preprint arXiv:2301.09211, 2023.
  19. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?i d=nZeVKeeFYf9. ↗
  20. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. Opt-iml: Scaling language m… — OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b. arXiv preprint arXiv: 2310.06825, 2023. find
  22. Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew William Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance, 2023. URL https://openreview.net/forum?id=ZAzSf9pz Cm. ↗
  23. Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings… find
  24. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Opena… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  25. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. arXiv preprint arXiv: 2305.15011, 2023.
  26. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, D…
  27. S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. International Conference on Machine Learning, 2023. doi: 10.485 50/arXiv.2301.13688.
  28. OpenAI. Gpt3.5, November 2022. URL https://platform.openai.com/docs/model s/gpt-3-5. ↗
  29. Maja Popović. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pp. 612–618, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4770. URL https://aclanthology.org/W17-4770. ↗
  30. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020… find
  31. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6399. URL https://doi.org/10.1609/aaai.v34i05. 6399. — [doi:10.1609/aaai.v34i05.6399](https://doi.org/10.1609/aaai.v34i05.6399) ↗
  32. SarvamAI. Openhathi series: An approach to build bilingual llms frugally, December 2023. URL https://www.sarvam.ai/blog/announcing-openhathi-series. ↗
  33. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  34. Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 2307.09288, 2023.
  35. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:… find
  36. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. Polylm: An open source polyglot large language model. arXiv preprint arXiv: 2307.06… find
  37. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Ita… find
  38. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023a.
  39. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv: 2306.05685, 2023b.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gala-2024-airavata-introducing-hindi-instruction-tuned/