📄 Creating a massively parallel Bible corpus

Summary

Researchers who compare many languages need parallel text, but the existing multilingual corpora either cover few languages (Europarl has 21, mostly European) or are hard to obtain. Mayer and Cysouw address this by building a massively parallel corpus from Bible translations, the most widely translated text in history. They gathered translations from PNGscriptures, Bible.is, Scripture Earth, Unboundbible and a collection by Östen Dahl, giving 994 unique translations under 837 ISO-639-3 codes. Every text was stripped to the bare verses, tokenized by separating non-letter characters, Unicode-normalized to NFC and then manually corrected. Verses are aligned through a numeric ID scheme. Copyright prevents full release, so the authors publish only the Gospel of Mark openly. For each complete translation they distribute a word-form frequency list and a sparse word-by-verse matrix, which keeps co-occurrence information while discarding word order. The result is a uniformly formatted resource that makes verse-level comparison across hundreds of languages practical.

Key points

• The first release has 994 translations with 837 ISO-639-3 codes; the codes were checked with diffs and trigram similarity. Several hundred more translations were awaiting processing.
• The translations contain 41,964 distinct verse numbers, including apocrypha, against 31,102 in the King James Version. Mark 1:7 is the most widely covered verse, present in 976 translations.
• Coverage varies widely across translations:
  • Verses per translation average 10,707 (SD 7,727), ranging from 677 in Wedau [wed] to 36,986 in the English King James Version.
  • Words per translation average about 409k.
  • Vocabulary size averages about 21k types.
• Each translation is packaged in three files: a .txt file with one verse per line (ID, TAB, text), a .wordforms file listing word forms alphabetically with their frequencies, and a .mtx file holding a sparse word × verse matrix in Matrix Market format.
• File names follow BCP 47, using the pattern ISO-x-bible-TRANSLATION-VERSION. Each verse also gets its own URL on paralleltext.info. Only Mark is freely downloadable.

Interesting ideas

• I1. Arithmetic-friendly verse IDs. An 8-digit ID (2 digits book, 3 chapter, 3 verse) lets you select a portion with a simple numeric range, such as IDs above 40,000,000 for the New Testament. The same IDs can serve directly as matrix column indices, which suits any aligned corpus with a hierarchical structure.
• I2. Sharing data without word order. Releasing only which words occur in which verse yields a "bag of sentences" that is arguably not copyright-protected yet still supports co-occurrence and alignment statistics. This is a useful pattern when the full text of a restricted corpus cannot be shared.
• I3. Empty versus missing verses. An ID with empty content means the verse was merged with a previous one, while an absent ID means the verse was never translated. Keeping this distinction preserves alignment information that is easily lost.
• I4. Versioned identifiers with retained old versions. The version tag is part of the identifier and earlier versions stay available. This allows errors to be fixed while citations and experiments that refer to older releases remain reproducible.
• I5. Human review for punctuation that is really orthography. Tokenizing by Unicode category fails when a symbol stands for a sound, such as an apostrophe-like mark used for the glottal stop in Arifama-Miniafia. The authors' experience suggests that low-resource orthographies need manual tokenization checks.