Elamur Research⌕
☰
home / papers / 2020 / Basic Language Resources for 31 Languages (Plus English): The LORELEI Representative and Incident Language Packs

Basic Language Resources for 31 Languages (Plus English): The LORELEI Representative and Incident Language Packs

Paper · 2020 · Jennifer Tracey and Stephanie Strassel

Summary

The DARPA LORELEI program needed language technology that could be moved quickly to an unfamiliar low-resource language during a crisis, such as a natural disaster. Systems would have almost no training data for that language. This paper from the Linguistic Data Consortium describes the data built to support and test such systems: 31 language packs plus partial English annotation. Twenty-four Representative Language (RL) packs, chosen to span many families, typologies and regions, were built for training and cross-lingual transfer research. They contain:

  • large monolingual and parallel corpora
  • entity, semantic and situation-frame annotation
  • lexicons and grammatical sketches
  • basic processing tools

Seven Incident Language (IL) packs served only as surprise test languages. Each came with a small adaptation set and held-out reference translations and annotations. The authors explain how text was found, translated and annotated, and how the pack design changed over the program. Together the packs hold over 2.5 billion monolingual words and 60 million parallel words. They are being released cheaply through the LDC catalog, which makes them a broad, uniform benchmark for low-resource NLP.

Key points

  • RL languages: Twenty-one were developed within the program (Swahili, Zulu, Yoruba, Akan, Wolof, Somali, Amharic; Arabic, Farsi, Hindi, Bengali, Tamil; Mandarin, Thai, Vietnamese, Tagalog, Indonesian; Russian, Ukrainian, Hungarian, Spanish), and three more, Hausa, Turkish and Uzbek, predate the program.

  • IL languages: Ilocano, Kinyarwanda, Odia, Oromo, Sinhala, Tigrinya and Uyghur. Each IL was paired with a related RL, for example Ilocano with Tagalog.

  • RL pack targets:

    • at least 2M monolingual words and at least 900K parallel words
    • 100Kw of English translated into the RL
    • 75Kw of named-entity annotation
    • 25Kw each of full entity/coreference, light semantic role labeling and situation frames
    • a lexicon of at least 10K lemmas
    • tools: segmenter, tokenizer, NE tagger and transliterator

    NP chunking was later dropped and entity linking was added.

  • Parallel text pipeline: web harvesting used BITS to find candidate document pairs and Champollion to align sentences. Translation was sourced as found text first, then crowd work, then professional translation to fill any remaining gap.

  • Overall scale: about 100 grammatical/lexical resources and 3 million annotation decisions made by more than 250 native speakers.

Interesting ideas

  • I1. The identity of each test language was kept secret until evaluation began, and the first scoring checkpoint came only 24 hours later, so systems were measured on real transfer rather than on tuning to the target.
  • I2. During web scouting, annotators tagged each document with metadata such as genre, incident type, location, eyewitness presence, how specific it was and its topics, and later tasks used these tags to choose data.
  • I3. Translations were sourced in a cost cascade: free found data first, then cheaper crowd work, and expensive professional effort only for the remainder.
  • I4. When one annotation task showed low inter-annotator agreement, the team kept the task and scored against several human references instead of redesigning it.
  • I5. In later evaluations, each IL test set came with a matching English set scored against English references, so the same content was judged in two languages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Keep a sealed set of genres or text sources that no repair rule or training run ever sees, and open it only for evaluation as a check against overfitting.
  • I2 (answer): Tag fables and school texts with similar properties (genre, number of characters, event density) so test sets can be assembled deliberately rather than at random.
  • I3 (learn): Take as much silver data as possible from agreeing witness parsers, and send only the leftover disagreements to costlier LLM teacher tiers.
  • I4 (answer): For open questions on tales, accept a set of valid gold answers so that legitimate interpretive variation does not count as error.
  • I5 (back): Score the text → graph → text compressor and meta-world translation on parallel inputs, checking that the meaning graphs built from both sides match.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Jennifer Tracey and Stephanie Strassel. 2020. Basic Language Resources for 31 Languages (Plus English): The LORELEI Representative and Incident Language Packs. Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tracey2020basic,
  title = {{Basic Language Resources for 31 Languages (Plus English): The LORELEI Representative and Incident Language Packs}},
  author = {Jennifer Tracey and Stephanie Strassel},
  booktitle = {Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)},
  year = {2020},
  url = {https://aclanthology.org/2020.sltu-1.39/}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M. and Schneider, N. (2013). Abstract Meaning Representation for Sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse.
  2. Imran, M., Castillo, C., Lucas, J., Meier, P., Vieweg, S. (2014). AIDR: Artificial Intelligence for Disaster Response. In: WWW’14 Companion. International World Wide Web Conference Committee (IW3C2) find
  3. Kirov, C., Cotterell, R., Sylak-Glassman, J., Walther, G., Vylomova, E., Xia, P., Faruqui, M., Mielke, S., McCarthy, A., Kübler, S., Yarowsky, D., Eisner, J., Hulden, M. (2018). UniMorph 2.0: Universal Morphology. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC’18), Miyazaki, Japan, May 7-12. European Language Resource Association (ELRA).
  4. Kiss, T., Strunkt, J. (2006). Unsupervised multilingual sentence boundary detection. Computational Linguistics 32: 455-525. ↗
  5. Kulick, S., Bies, A., Maamouri, M. (2010). Consistent and Flexible Integration of Morphological Annotation in the Arabic Treebank. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valetta, Matlta, May 17-23. European Language Resource Association (ELRA). find
  6. Kulick, S., Bies, A. (2016). Rapid Development of Morphological Analyzers for Typologically Diverse Languages. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portoroz, Slovenia, May 23-28. European Language Resource Association (ELRA). find
  7. Li, X., Ge, N., Grimes, S., Strassel, S., Maeda, K. (2010). Enriching Word Alignment with Linguistic Tags. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valetta, Matlta, May 17-23. European Language Resource Association (ELRA). find
  8. Ma, X. (2006). Champollion: A Robust Parallel Text Sentence Aligner. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy, May 22-28. European Language Resource Association (ELRA). find
  9. Ma, X., Liberman, M. (1999). BITS: A Method for Bilingual Text Search over the Web. In Proceedings of Machine Translation Summit VII, Singapore, September 13-17. find
  10. META-NET. 2012. META-NET White Paper Series, https://link.springer.com/bookseries/10412, accessed March 20, 2020. ↗
  11. Mott, J., Bies, A., Strassel, S., Kodner, J., Richter, C., Xu, H., Marcus, M. (2020). Morphological Segmentation for Low Resource Languages. find
  12. NIST LoReHLT Evaluations Website. https://www.nist.gov/itl/iad/mig/lorehlt-evaluations. Retrieved February 14, 2020. ↗
  13. Palmer, M., Gildea, D., Kingsbury, P. (2005). The Proposition Bank: A Corpus Annotated with Semantic Roles. Computational Linguistics Journal, 31:1. find
  14. Strassel, S., Bies, A., Tracey, J. (2017). Situational Awareness for Low Resource Languages: the LORELEI Situation Frame Annotation Task. First Workshop on Exploitation of Social Media for Emergency Relief and Preparedness (SMERP), Aberdeen, Scotland, April 8-13. find
  15. Strassel, S., Tracey, J. (2016). LORELEI Language Packs: Data, Tools, and Resources for Technology Development in Low Resource Languages. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portoroz, Slovenia, May 23-28. European Language Resource Association (ELRA).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tracey-2020-basic-language-resources-31-languages/