Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck (2024 · 9 citations)
Small language models are cheaper to train and serve. However, the Pythia suite showed that when they are trained on very large corpora, they can saturate: late in pretraining, their loss rises and then stagnates. The authors attribute this to the softmax bottleneck. A linear prediction head built on a small hidden…
Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level Annotation (2026)
Scientific articles in PubMed Central are internally uneven. Clinical case descriptions and pedagogical review passages appear alongside methods boilerplate and low-information text. Filtering whole documents, as FineWeb-Edu does for web pages, therefore throws away useful material. The authors instead label PMC Open…

All works

3 works of theirs in our library, the most cited first; 2 with a page here, the others link their original until we summarise them.

Why do small language models underperform? Studying Language Model Saturation via the Softmax BottleneckarXiv:2404.07647Nathan Godey, Eric de la Clergerie, Benoit Sagot2024 · 9 citations
Gaperon: A Peppered English-French Generative Language Model Suite ↗Findings of the Association for Computational Linguistics: ACL 2026 · no summary yetNathan Godey et al.2026 · 5 citations
Biomed-Enriched: Data-Efficient Biomedical Pretraining via Paragraph-Level AnnotationFindings of the Association for Computational LinguisticsRian Touchent, Nathan Godey, Éric Villemonte de la Clergerie2026

Co-authors here

3 co-authors with a page, the most papers together first.

Where their works here were published, the most works first, with the editions of those that have a page.

Papers of our library cite their works 14 times (of the 149,439 papers whose bibliographies we read); 5 citing papers with a page here →.

Updated 1 November 2026