🧑‍🔬 Michael Pieler

Michael Pieler works on large language models, with papers here that span 2022 to 2024. Two questions run through them: how to build and openly release autoregressive Transformer models, and how to adapt an existing model to a language it handles poorly. The methods are mostly about pretraining: choosing and cleaning corpora, balancing data sources and evaluating the resulting models. This places the work under language models & pretraining and, in its more recent part, Semitic languages.

The most cited paper is GPT-NeoX-20B (2022). It presents a dense Transformer of the decoder-only kind with 20 billion parameters, which EleutherAI trained on the Pile corpus. The weights were made freely downloadable so that people without access to commercial systems could examine how large models behave, for instance in work on safety, interpretability or scaling. When the paper was submitted, no larger dense autoregressive model was openly available in this way.

The newer work turns from building a model from scratch to adapting one to a new language. Arabic Stable LM (2024) continues pretraining the 1.6B-parameter Stable LM 2 on English mixed with cleaned Arabic web, news and e-book text. A chat version adds synthetic Arabic dialogues that an LLM produced by rephrasing existing material. The authors measure how heavily the tokenizer splits Arabic words and correct the sampling mix for the token counts this inflates. They also re-derive quality-filter thresholds from statistics of Arabic itself. The paper reports that these compact models match or outperform much larger Arabic-focused ones.

Pieler is currently at Edison Scientific Inc. in San Francisco, after an earlier affiliation with New York University in 2023. Co-authors on the papers here include Zaid Alyafeai, first author of the Arabic work, as well as Stella Biderman, Leo Gao and Jason Phang.

Michael Pieler has 3 papers in our library, in 2022–2024, most of them on Language models & pretraining, Semitic languages; often with Aaron Mueller, Alexis Ross, Alicia Parrish.