📄 LORELEI Language Packs: Data, Tools, and Resources for Technology Development in Low Resource Languages

Summary

The paper describes how the Linguistic Data Consortium (LDC) builds text language packs for DARPA's LORELEI program. LORELEI aims to make language technology work for low-resource languages and to retrain it quickly for a new language during an emergency, such as a natural disaster or an epidemic. LDC is producing 35 packs: 23 Representative Languages chosen for typological breadth, and 12 Incident Languages, whose evaluation languages stay secret until testing starts.

Each Representative Language pack contains:

• web-harvested monolingual text,
• parallel text with English,
• several annotation layers (entities, NP chunking, simple semantics, morphology/POS, situation frames),
• processing tools, lexicons and grammatical sketches.

The main design choice is to treat all packs as one coordinated resource. Specifications, tag sets and guidelines are shared across languages and adapted locally where needed. This makes every pack useful on its own and also as part of a multilingual collection for cross-language transfer. Annotation tasks are built for non-linguist native speakers working under tight deadlines. At the time of writing, the Turkish, Hausa and Uzbek packs had been delivered.

Key points

• Languages: 23 Representative Languages and 12 Incident Languages, 35 packs in total. The first three packs (Uzbek, Turkish, Hausa) came from the DARPA BOLT program and do not fully meet the final LORELEI requirements.
• Monolingual text: at least 2 million words per Representative Language, split into 50% news, 40% blogs/forums and 10% microblogs. At least half of the material sent on to translation and annotation must be in-domain.
• Parallel text:
  • 900,000 words per language are translated into English.
  • A fixed English Core of about 100,000 words is translated into every language. It includes domain news, general news, a phrasebook and a CMU elicitation corpus.
  • About 2,000 words of the English Core are annotated in each language.
• Tools and pipeline: collection uses WebCol, with an IPR review of every harvested source or document. BITS finds parallel text, Champollion aligns sentences, and Google CLD2 performs language ID. Translations come from found data, crowdsourcing and professional translators.
• Quality control and release: packs are checked by LDC and then independently by UMD CASL. They are free to performers and LDC members and available to others for a small fee. The first packs were expected in the LDC Catalog in late 2016.

Interesting ideas

• I1. Separate text from structure. Store the raw linguistic content as plain UTF-8 and keep document structure and metadata in a stand-off XML layer. Recombine the two only for release. This keeps messy web markup from contaminating the text and makes reprocessing easier for any web-crawled corpus.
• I2. Design annotation for non-experts. Break each annotation task into a chain of small yes/no or choice decisions, and mirror those decisions in the guidelines and the interface. This lets untrained native speakers produce usable labels quickly, which helps wherever trained linguists are scarce.
• I3. Share a common annotated slice across languages. Annotating the same small English Core translation in every language creates a parallel annotated subset. That subset supports cross-lingual comparison and projection at a modest cost.
• I4. Use language ID for source auditing, not strict filtering. Treat automatic LID as a way to spot systematic problems with a data source, and accept some mixing of languages or scripts (e.g., Uzbek in Cyrillic, Latin or Perso-Arabic). This is realistic for informal or multi-script data.
• I5. Keep the test languages secret. Holding back the evaluation languages until test time measures real adaptation speed rather than prior tuning. The same idea applies to any evaluation of transfer methods.