Elamur Research⌕
☰
home / papers / 2014 / The AMARA Corpus: Building Parallel Language Resources for the Educational Domain

The AMARA Corpus: Building Parallel Language Resources for the Educational Domain

Summary

Machine translation of lecture videos suffers from a lack of in-domain training data, even as MOOC platforms keep publishing large volumes of mostly English material. The authors address this by building the AMARA corpus from volunteer-made subtitles on the Amara platform, which Khan Academy, Udacity, TED and other educational providers use. They crawled the site, kept only educational videos using team information and video metadata, and discarded empty, incomplete or wrongly labelled subtitles. They then aligned the subtitles at the segment level, using strict timestamp matching followed by a dictionary-assisted cascaded alignment that recovers out-of-sync segments. The release (version 1.4) covers 20 languages, giving 20 monolingual and 190 parallel corpora. For 14 language pairs, it also provides development and test sets that are aligned jointly and at the whole-document level, along with baseline SMT results. The error analysis shows that this spoken, formula-heavy content is often fairly easy to translate but hard to evaluate fairly. The resource matters most for under-served languages, where volunteers cannot keep up with new content.

Key points

  • Crawl: over 121K translated documents for 43K videos in more than 160 languages, collected in August 2013 with Amara's cooperation.
  • Domain filtering: this step left 23.1K videos and 44.6K subtitles, drawn from Khan Academy, Coursera, Udacity and other education or science sources.
  • Validation: about 30K empty or incomplete subtitles were dropped, and the Cybozu language detector caught subtitles in the wrong language. This left 34K subtitles for 12.2K videos.
  • Corpus size: the release is limited to the 20 best-resourced languages, from English down to Hindi, Thai and Danish, totalling about 5.6M segments and 47.1M tokens.
  • Alignment: roughly 75% of segments already share identical timestamps across languages.
    • Cascaded synchronization, which uses a Hunalign-derived bilingual lexicon, yielded roughly 2.5–2.9× more aligned data than strict matching in the reported comparison.
    • The OOV rate stayed under 3% for most language pairs.

Interesting ideas

  • I1. Two-stage alignment: first align only the trivially parallel portion (identical IDs and timestamps). Then learn a bilingual lexicon from that clean subset and use it to align the leftover material. This bootstrapping approach suits any semi-structured parallel source, such as subtitles, localized documentation or parallel web pages.
  • I2. Stricter thresholds for dev/test data: training data accepts all aligned segments, but development and test sets keep only high-confidence alignments. This lets you use as much training data as possible while keeping evaluation data reliable.
  • I3. Filtering by metadata: the authors selected the domain using who produced the content (teams) and platform categories, not text classifiers. This is a cheap, transparent way to carve a domain out of a mixed crowd-sourced collection.
  • I4. Checking claimed languages: running a language detector over user-labelled data removes mislabelled items before they corrupt alignment. This simple check is worth applying to any crowd-sourced multilingual dataset.
  • I5. Normalizing notation before scoring: the paper notes that BLEU unfairly penalizes correct output when symbols and spoken forms differ, such as "<" versus "less than" or digits versus words. Normalizing these forms beforehand, or using a paraphrase-aware metric like METEOR, makes evaluation fairer for technical or math-heavy text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

79 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abdelali2014amara,
  title = {{The AMARA Corpus: Building Parallel Language Resources for the Educational Domain}},
  author = {Abdelali, Ahmed and Guzman, Francisco and Sajjad, Hassan and Vogel, Stephan},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/47mc9uc5rj4q},
  url = {https://doi.org/10.63317/47mc9uc5rj4q}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 19(2):263–311. find
  2. Cettolo, M., Girardi, C., and Federico, M. (2012). WIT 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation, EAMT ’12, Trento, Italy. find
  3. Durrani, N., Schmid, H., and Fraser, A. (2011). A joint sequence translation model with integrated reordering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, USA. find
  4. Fügen, C., Kolss, M., Bernreuther, D., Paulik, M., Stücker, S., Vogel, S., and Waibel, A. (2006). Open domain speech recognition & translation: Lectures and speeches. In Acoustics, Speech and Signal Processing, ICASSP ’06, Toulouse, France. find
  5. Fügen, C., Waibel, A., and Kolss, M. (2007). Simultaneous translation of lectures and speeches. Machine Translation, 21(4):209–252. find
  6. Gale, W. A. and Church, K. W. (1993). A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75–102. find our card
  7. Green, S. and DeNero, J. (2012). A class-based agreement model for generating accurately inflected translations. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, ACL ’12, Jeju Island, Korea. find
  8. Green, S., Heer, J., and Manning, C. D. (2013). The efficacy of human post-editing for language translation. In ACM Human Factors in Computing Systems, CHI ’13, Paris, France. find
  9. Guzman, F., Sajjad, H., Abdelali, A., and Vogel, S. (2013). The AMARA corpus: Building resources for translating the web’s educational content. In Proceedings of the International Workshop on Spoken Language Translation, IWSLT ’13, Heidelberg, Germany. find
  10. Heafield, K. (2011). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, WMT ’11, Edinburgh, UK. find our card
  11. Hopkins, M. and May, J. (2011). Tuning as ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, Edinburgh, Scotland, United Kingdom. find
  12. Jansen, D., Alcala, A., and Guzman, F. (2014). Amara: A sustainable, global solution for accessibility, powered by communities of volunteers. In Proceedings of the 16th International Conference on Human-Computer Interaction, HCII ’14, Heraklion, Greece. find
  13. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, HLT-NAACL ’03, Edmonton, Canada. find
  14. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Asso… find
  15. Kumar, S. and Byrne, W. (2004). Minimum bayes-risk decoding for statistical machine translation. In Proceedings of HLT-NAACL 2004: Main Proceedings, Boston, Massachusetts, USA. find our card
  16. Lavie, A. and Denkowski, M. J. (2009). The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2-3):105–115. find
  17. Mostefa, D., Moreau, N., Choukri, K., Potamianos, G., Chu, S. M., Tyagi, A., Casas, J. R., Turmo, J., Cristoforetti, L., Tobia, F., et al. (2007). The CHIL audiovisual corpus for lecture and meeting analysis inside smart rooms. Language Resources and Evaluation, 41(3-4):389–407. find
  18. Nakov, P., Guzman, F., and Vogel, S. (2012). Optimizing for sentence-level BLEU+1 yields short translations. In Proceedings of the 24th International Conference on Computational Linguistics, COLING ’12, Mumbai, India. find
  19. Nakov, P., Guzman, F., and Vogel, S. (2013). A tale about PRO and monsters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria. find
  20. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL ’02, Philadelphia, USA. find our card
  21. Paul, M., Federico, M., and Stüker, S. (2010). Overview of the IWSLT 2010 evaluation campaign. In Proceedings of the International Workshop on Spoken Language Translation, IWSLT ’10, Paris, France. find
  22. Silvestre-Cerdà, J. A., del Agua, M. A., Garcés, G., Gascó, G., Giménez, A., Martı́nez, A., Pérez, A., Sánchez, I., Serrano, N., Spencer, R., Valor, J. D., Andrés-Ferrer, J., Civera, J., Sanchis, A., and Juan, A. (2012). TransLectures. In Online Proceedings of Advances in Speech and Language… find
  23. Tiedemann, J. (2008). Synchronizing translated movie subtitles. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC ’08, Marrakech, Morroco. find
  24. Tiedemann, J. (2012). Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation, LREC ’12, Istanbul, Turkey. find
  25. Varga, D., Németh, L., Halácsy, P., Kornai, A., Trón, V., and Nagy, V. (2005). Parallel corpora for medium density languages. In Proceedings of the Recent Advances in Natural Language Processing, RANLP ’05, Borovets, Bulgaria. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdelali-2014-amara-corpus-building-parallel-language/