語彙置換継続事前学習による日英バイリンガルモデルの構築と評価

Summary

The authors, from Ricoh, addressed the following problem: the English-centric LLM Llama 2 has a tokenizer that is poorly suited to Japanese. On average, one token covers less than one Japanese character, which makes training and inference in Japanese more expensive and means the context window holds less text. To fix this, they continued pretraining Llama 2 13B Chat using a vocabulary replacement method. The vocabulary size remained the same, but rarely used tokens were replaced with Japanese ones. The new tokens were obtained via BPE in SentencePiece trained on a Japanese–English corpus. The model was then trained on public Japanese and English data using two types of curriculum learning: the first takes data quality into account, the second the language ratio. The resulting model was named Ricoh-13B. On the Japanese llm-jp-eval benchmarks, it achieved the highest average score among the compared 13B models with Japanese-adapted tokenizers. At the same time, English performance degraded somewhat due to forgetting. The work shows that, with relatively modest resources, an English-centric model can be efficiently adapted to a lower-resource language. This also applies to Ukrainian.

Key points

• Tokenizer: the vocabulary was kept at 32,000 tokens. The Japanese portion grew from roughly 1k to 17k, while the English portion shrank from 28k to 14k. LPT (average number of characters per token) for Japanese rose from 0.85 to 2.01, while for English it barely changed: 4.01 → 3.99.
• Data: Japanese corpora Wikipedia (ja), CC100 (ja), OSCAR (ja) and mC4 (ja). English: subsets of RedPajama v1: Wikipedia (en), Book, Stack Exchange and C4.
• Curriculum: by quality, the order is as follows: first a small amount of high-quality data, then a large mass of lower-quality data, and finally high-quality data again. By language, English dominates at the beginning, and Japanese in the second half of training.
• Training: 64 AWS Trn1 nodes (trn1.32xlarge) on AWS Neuron NeMo Megatron, batch size of 4M tokens, learning rate 8.0×10⁻⁵, AdamW optimizer. At each data switch, the learning rate was warmed up again (warmup).
• Results: on llm-jp-eval, the average score increased by 8–9 points compared with Llama 2 13B (58.9) and Llama 2 13B Chat (57.9). Ricoh-13B outperformed PLaMo-13B, LLM-jp-13B, Swallow-13B and ELYZA-japanese-Llama-2-13b-fast. In English (lm-evaluation-harness), results are lower: GLUE 62.1 versus 63.0 for Llama 2 13B, Open LLM Leaderboard average 51.9 versus 55.4.