📄 SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling

Summary

Smaller benchmarks for word-level language modeling are not good stand-ins for larger ones. In small corpora each word type occurs only a few times. As a result, architectures that lead on small sets, such as LSTMs on WikiText-2, often lose on large sets, where Transformers dominate WikiText-103. This weakens cheap experimentation such as architecture search. The author builds SimpleBooks from Project Gutenberg. Books are selected for having a small vocabulary relative to their length, which mostly yields children's literature. Near-duplicate books are then removed. The result is SimpleBooks-92, with 92M tokens but only about 98K word types, and SimpleBooks-2, a 2M-token training subset that shares the same validation and test sets. Average token frequency approaches that of 1Billion, and OOV rates are very low. Transformer-XL beats AWD-LSTM on both sizes, so the small version mirrors large-data behaviour. The models also need far fewer parameters than their WikiText counterparts. A skip-gram transfer test from simple to normal English shows faster early learning but similar final loss.

Key points

• Selection and splits. The corpus comes from 1,573 Gutenberg books, chosen from about 39K candidates.
  • Selection used a vocabulary-to-length ratio that includes a squared term and avoids favouring long books, with a threshold of 0.0012.
  • Books were deduplicated greedily, discarding any book with ≥50% 8-gram overlap with books already kept.
  • Validation and test sets each consist of 5 books, about 200K tokens each.
• Corpus statistics.
  • SimpleBooks-92: about 91.5M tokens, vocabulary 98,304, FREQ 931.4, OOV 0.11%.
  • SimpleBooks-2: about 2.2M tokens, vocabulary 11,492, FREQ 195.43.
• Perplexity results. Test perplexity on SB-92 is 8.92 for Transformer-XL versus 20.64 for AWD-LSTM. On SB-2 it is 16.41 versus 16.78.
• Efficiency on SB-2. Transformer-XL used 19.7M parameters and about 80 epochs, while AWD-LSTM used 29.2M parameters and about 200 epochs.
• Comparison with WikiText-103.
  • With near-SOTA settings, SB-92 cuts parameters by 45.3% for Transformer-XL and 39.7% for AWD-LSTM compared with WikiText-103, mostly through smaller embedding layers.
  • Despite having only 36.7% of WikiText-103's vocabulary, SB-92 covers 92% of its tokens (93% uncased).

Interesting ideas

• I1. Using average occurrences per word type (FREQ) as a diagnostic shows whether a small dataset behaves like a big one, which is more informative than raw token count.
• I2. Greedy near-duplicate removal works by going from the largest document to the smallest and dropping any whose 8-gram overlap with the kept set reaches half. This simple rule keeps editions and reprints from inflating a corpus.
• I3. Children's books serve as a naturally low-vocabulary, long-context source of English, filtered by a lexical-density ratio rather than by manual labelling.
• I4. For warm-start transfer, embeddings learned on simple text initialise the shared rows, and unseen words are drawn uniformly within the learned value range.
• I5. Before measuring vocabulary coverage, re-tokenise the target corpus to match conventions (e.g. "do n't" instead of "don 't"), because otherwise coverage figures are misleading.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): compute FREQ for silver training sets and small evaluation sets to judge whether results on a small sample should transfer to the full data.
• I2 (answer): apply it to collections of fables and tales so that retold variants do not leak between practice and test stories.
• I3 (tree): use such books as an easier first tier for gparse training or for agreement among witness parsers before moving to harder prose.
• I4 (tree): pre-train gparse's lexical layer on simplified text, then expand to full vocabulary without a scale mismatch for new words.
• I5 (tokens): normalise contraction splitting across treebanks and input text so that lemma and coverage statistics in the en crate are comparable.