Elamur Research⌕
☰
home / papers / 2023 / A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models

A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models

Paper · 2023 · Hyunwoong Ko, Kichang Yang, Minho Ryu et al.

Summary

Multilingual models such as mBERT, XGLM and BLOOM learn mostly from English-heavy corpora and tend to underperform in other languages. As the first output of EleutherAI's Polyglot project, the authors built Polyglot-Ko, a family of open Korean decoder-only models. Together with TUNiB, they assembled 1.2TB of Korean text from blogs, news, the Modu corpus, patents, Q&A, comments and other sources, which filtering reduced to 863GB. Cleaning removed duplicates, PII, leftover HTML, very short texts and repeated characters. Four models (1.3B, 3.8B, 5.8B and 12.8B parameters) were trained with the GPT-NeoX codebase on Stability AI hardware. All four share a morpheme-aware tokenizer. On the KOBEST benchmark, under zero- and few-shot settings, Polyglot-Ko generally outperformed other public billion-scale Korean models, and the 12.8B model scored best on most tasks. Scores also rose with compute. The 12.8B model was presented as the largest openly released Korean LM licensed for commercial use, which makes it a practical base for Korean NLP.

Key points

  • Data and tokenizer: 1.2TB of raw Korean data became 863GB after preprocessing, with blog posts making up most of it (~682GB). The tokenizer is a morpheme-aware Byte-Level BPE built on MeCab analysis, with a vocabulary of 30,003.
  • Training setup and token counts: The models ran on 256 A100 GPUs, with token counts of 213B (1.3B), 219B (3.8B), 172B (5.8B) and 167B (12.8B).
  • Early stopping in the smaller models: The 1.3B and 3.8B models began producing repetitive output when their loss dropped sharply near the first epoch boundary, around 100k steps. The authors therefore kept checkpoints from before that point.
  • Evaluation: KOBEST covers COPA, HellaSwag, BoolQ, SentiNeg and WiC. The baselines were ko-gpt-trinity-1.2B, KoGPT and XGLM-7.5B. KoGPT did win zero-shot HellaSwag (0.5590), and the authors found that a revised SentiNeg prompt stabilized zero-shot results.
  • Known flaws and future plans: Newlines were mistakenly stripped during preprocessing, which lost document structure. Planned work includes a 40B Korean model plus East-Asian and Romance multilingual models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Jiwung Hyun, Sungho Park and Kyubyong Park. 2023. A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models. arXiv:2306.02254.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.02254 · arXiv 2306.02254

@misc{ko2023technical,
  title = {{A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models}},
  author = {Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Jiwung Hyun, Sungho Park and Kyubyong Park},
  year = {2023},
  doi = {10.48550/arxiv.2306.02254},
  eprint = {2306.02254},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.02254}
}

References

19 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. 2021. GPT-NeoX: Large Scale Autoregressive Language Modeling in… find
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — PaLM 2 Technical Report
  3. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across trainin… find
  4. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregr…
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  6. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. find
  7. Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  9. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. find
  10. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
  11. Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. 2022. Kobest: Korean balanced evaluation of significant tasks. find
  12. Ildoo Kim, Gunsoo Han, Jiyeon Ham, and Woonhyuk Baek. 2021. Kogpt: Kakaobrain korean(hangul) generative pre-trained transformer. https://github.com/kakaobrain/kogpt. ↗
  13. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — Few-shot Learning with Multilingual Language Models
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  16. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  17. MosaicML NLP Team. 2023. Introducing mpt-7b: A new standard for open-source, ly usable llms. Accessed: 2023-03-28. find
  18. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
  19. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ko-2023-technical-report-polyglot-ko-open/