Elamur Research⌕
☰
home / papers / 2023 / TigerBot: An Open Multilingual Multitask LLM

TigerBot: An Open Multilingual Multitask LLM

Paper · 2023 · Ye Chen, Wei Cai, Liangmin Wu et al.

Summary

Open-source LLMs such as Llama-2 are strong in English but cover Chinese and other East Asian languages poorly, and building competitive models is usually expensive. Tiger Research addresses both problems with TigerBot, a family of base and chat models at 7B, 13B, 70B and 180B parameters. The three smaller models continue pretraining from Llama-2, and the largest builds on BLOOM. The team assembled a pretraining mix of roughly 500B tokens from 25 sources, with Chinese and English in near-equal shares. They cleaned it with format rules, deduplication and safety classifiers. They also enlarged the Llama-2 vocabulary for Asian scripts, mixed a small share of instruction data into pretraining, and aligned the chat models with SFT, rejection sampling and DPO. A customized Megatron-DeepSpeed fork achieved higher training throughput than Llama-2 reported. The models beat Llama-2 by about 6% in English and 20% in Chinese, and they rank highly on OpenCompass and CLiB. The release matters because the weights, tools and methodology are open and were produced at modest cost. The authors also share practical lessons from search, role-playing and hardware deployments.

Key points

  • Data: pretraining used about 470–500B tokens (≈1.8TB), filtered down from 5.6TB. Sources include WuDao, WanJuan, C4, RefinedWeb, BookCorpus, Clibrary, GitHub, arXiv and wikis. About 10% of the data was removed as low quality.
  • Tokenizer: a BPE SentencePiece tokenizer was trained on 100GB of Chinese, Japanese and Korean text, then merged into the Llama-2 tokenizer. This raised the vocabulary from 32k to just under 65k. TigerBot-180B keeps BLOOM's 250k vocabulary.
  • Alignment data: 5M instruction examples for SFT and 15k human comparison pairs (10k for rejection sampling, 5k for DPO), spanning about 100 task subcategories.
  • Efficiency: TigerBot-13B processed 1,645 tokens per GPU-second on 32× A100-40G (TP=2, PP=8, DP=2). This is about 109% of Llama-2's rate. Total cost was under $2M for April–December 2023.
  • Release and applications: models are released under Apache-2.0, subject to the Llama-2 and BLOOM licenses. Applications include context windows up to 32k, search and document plug-ins, and role-playing that combines fine-tuning with RAG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin and Cong Fu. 2023. TigerBot: An Open Multilingual Multitask LLM. arXiv:2312.08688.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.08688 · arXiv 2312.08688

@misc{chen2023tigerbot,
  title = {{TigerBot: An Open Multilingual Multitask LLM}},
  author = {Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin and Cong Fu},
  year = {2023},
  doi = {10.48550/arxiv.2312.08688},
  eprint = {2312.08688},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.08688}
}

References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv:2305.13245 [cs.CL], 05 2023.
  2. Anthropic. Claude 2. https://www.anthropic.com/index/claude-2, 06 2023. ↗
  3. J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic parsing on freebase from question-answer pairs. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
  4. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — Language Models are Few-Shot Learners
  5. H. Chen, R. Pasunuru, J. Weston, and A. Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv:2310.05029 [cs.CL], 10 2023. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) find
  6. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457 [cs.AI], 03 2018.
  7. O. Contributors. Opencompass: A universal evaluation platform for foundation models. GitHub repository, 2023. find
  8. Y. Cui, Z. Yang, and X. Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv:2304.08177 [cs.CL], 04 2023.
  9. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv:2205.14135 [cs.LG], 05 2022. find
  10. Google. Sentencepiece. GitHub repository, 2023. find
  11. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.09685 [cs.CL], 06 2021.
  12. Huggingface. Text generation inference. GitHub repository, 2023. find
  13. Huggingface. Transformers. GitHub repository, 2023. find
  14. V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih. Dense passage retrieval for open-domain question answering. EMNLP 2020, 04 2020.
  15. C. Li, Z. Leng, C. Yan, J. Shen, H. Wang, W. MI, Y. Fei, X. Feng, S. Yan, H. Wang, L. Zhan, Y. Jia, P. Wu, and H. Sun. Chatharuhi: Reviving anime character in reality via large language model. arXiv:2308.09597 [cs.CL], 2023.
  16. Microsoft. Megatron-deepspeed. GitHub repository, 2023. find
  17. D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. arXiv:2104.04473 [cs.CL], 04 2021. find
  18. NVIDIA. Tensorrt open source software. GitHub repository, 2023. find
  19. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. ar…
  20. O. Peckham. Meta completes research supercluster, announces next-gen datacenter. HPCwire: https://www.hpcwire.com/2023/05/18/meta-completes-research-supercluster-announces-next-gen-datacenter/, 05 2023. ↗
  21. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. arXiv:2309.00071 [cs.CL], 09 2023. find
  22. S. Pichai. An important next step on our ai journey. https://blog.google/technology/ai/bard-google-ai-search-updates/, 02 2023. ↗
  23. O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv:2108.12409 [cs.CL], 08 2021. find
  24. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv:2305.18290 [cs.LG], 05 2023.
  25. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. arXiv:1910.02054 [cs.LG] and In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’20), 10 2019. find
  26. P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad. arXiv:1806.03822 [cs.CL], 06 2018.
  27. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguye… find
  28. N. Shazeer. Glu variants improve transformer. arXiv:2002.05202 [cs.LG], 02 2020. find
  29. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864 [cs.CL], 04 2021.
  30. A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv:1811.00937 [cs.CL], 11 2018.
  31. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. GitHub repository, 2023. find
  32. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv:2302.13971 [cs.CL], 02 2023.
  33. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  34. Turboderp. Exllamav2. GitHub repository, 2023. find
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA., 06 2017.
  36. J. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano. Recursively summarizing books with human feedback. arXiv:2109.10862 [cs.CL], 09 2021.
  37. W. Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz, M. Khabsa, H. Fang, Y. Mehdad, S. Narang, K. Malik, A. Fan, S. Bhosale, S. Edunov, M. Lewis, S. Wang, and H. Ma. Effective long-context scaling of foundation models. arXiv:2309.16039 [cs.C…
  38. Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv:2206.01861 [cs.CL], 06 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-tigerbot-open-multilingual-multitask/