Elamur Research
☰
home / papers / 2014 / Report on the 11th IWSLT evaluation campaign

Report on the 11th IWSLT evaluation campaign

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/2014.iwslt-evaluation.1.pdf

Summary

This report describes the 2014 (11th) evaluation campaign of the IWSLT workshop. The campaign benchmarks the core technologies of spoken language translation on TED and TEDx lectures. Three task families were offered:

  • ASR: speech-to-transcript in English, German and Italian, with Italian new this year.
  • SLT: speech-to-foreign-text.
  • MT: translation of clean transcripts.

The official SLT and MT directions paired English with French, German and Italian. Optional directions involved 12 further languages. Organizers supplied training and development data, mainly WIT 3 TED material plus WMT 2014, MultiUN and SETimes corpora. Each test period lasted about a week. All submissions were scored automatically: WER for ASR, BLEU/TER for translation. Teams also ran their systems on the 2013 test set so that year-over-year progress could be measured. For English-French and English-German MT, professional translators post-edited system outputs, and HTER was computed from these post-edits. Twenty-one teams took part. English ASR improved clearly over 2013, while TEDx German audio remained hard. The campaign provides a shared multilingual benchmark for lecture translation and tracks progress over time.

Key points

  • 21 participating sites submitted 76 primary runs: 15 ASR, 16 SLT and 45 MT.
  • Primary ASR WER on tst2014 ranged from 8.4–19.7% for English, 24.0–38.8% for German and 21.9–25.4% for Italian.
    • Two German TEDx talks exceeded 40% WER because of poorer recordings.
  • On tst2013, the best English WER fell from 13.5% to 10.6%, about a 21% relative gain in one year.
    • The organizers' ROVER combination of five English systems, used as SLT input, reached 8.2% WER.
  • Supplied resources included the WIT 3 TED corpus, WMT 2014 data, MultiUN, SETimes and LDC monolingual corpora.
  • Human evaluation used HTER on professional post-edits for the English-French and English-German MT tracks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cettolo2014report,
  title = {{Report on the 11th IWSLT evaluation campaign}},
  author = {Cettolo, Mauro and Niehues, Jan and Stüker, Sebastian and Bentivogli, Luisa and Federico, Marcello},
  booktitle = {Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign},
  year = {2014},
  url = {https://aclanthology.org/2014.iwslt-evaluation.1/}
}

References

41 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Akiba, M. Federico, N. Kando, H. Nakaiwa, M. Paul, and J. Tsujii, “Overview of the IWSLT04 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Kyoto, Japan, 2004, pp. 1–12. find
  2. M. Eck and C. Hori, “Overview of the IWSLT 2005 evaluation campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Pittsburgh, PA, 2005, pp. 1–22. find
  3. P. Michael, “Overview of the IWSLT 2006 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Kyoto, Japan, 2006, pp. 1–15. find
  4. C. S. Fordyce, “Overview of the IWSLT 2007 evaluation campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Trento, Italy, 2007, pp. 1–12. find
  5. M. Paul, “Overview of the IWSLT 2008 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Waikiki, Hawaii, 2008, pp. 1–17. find
  6. ——, “Overview of the IWSLT 2009 Evaluation Campaign,” in Proceedings of the sixth International Workshop on Spoken Language Translation, Tokyo, Japan, 2009, pp. 1–18. find
  7. M. Paul, M. Federico, and S. Stüker, “Overview of the IWSLT 2010 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Paris, France, 2010, pp. 3–27. find
  8. M. Federico, L. Bentivogli, M. Paul, and S. Stüker, “Overview of the IWSLT 2011 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, San Francisco, USA, 2011, pp. 11–27. find
  9. M. Federico, M. Cettolo, L. Bentivogli, M. Paul, and S. Stüker, “Overview of the IWSLT 2012 Evaluation Campaign,” in Proceedings of the International Workshop on Spoken Language Translation, Hong Kong, HK, 2012, pp. 11–27. find
  10. M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico, “Report on the 10th IWSLT Evaluation Campaign,” in Proceedings of the Tenth International Workshop on Spoken Language Translation (IWSLT 2013), Heidelberg, Germany, 2013. find
  11. M. Cettolo, C. Girardi, and M. Federico, “WIT 3 : Web Inventory of Transcribed and Translated Talks,” in Proceedings of the Annual Conference of the European Association for Machine Translation (EAMT), Trento, Italy, May 2012. [Online]. Available: http://hltshare.fbk.eu/EAMT2012/html/Papers/59.pdf ↗
  12. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna, “Findings of the 2014 Workshop on Statistical Machine Translation,” in Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimo… find our card
  13. M. Freitag, J. Wuebker, S. Peitz, H. Ney, M. Huck, A. Birch, N. Durrani, P. Koehn, M. Mediani, I. Slawik, J. Niehues, E. Cho, A. Waibel, N. Bertoldi, M. Cettolo, and M. Federico, “Combined Spoken Language Translation,” in Proceedings of the 11th International Workshop on Spoken Language Translation… find
  14. B. Babaali, R. Serizel, S. Jalalvand, D. Falavigna, R. Gretter, and D. Giuliani, “FBK @ IWSLT 2014 - ASR track,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  15. N. Bertoldi, P. Mathur, N. Ruiz, and M. Federico, “FBK’s Machine Translation and Speech Translation Systems for the IWSLT 2014 Evaluation Campaign,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  16. M. Beloucif, C.-K. Lo, and D. Wu, “Improving tuning against MEANT,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  17. Q. B. Nguyen, T. T. Vu, and C. M. Luong, “The Speech Recognition Systems of IOIT for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  18. K. Kilgour, M. Heck, M. Müller, M. Sperber, S. Stüker, and A. Waibel, “The 2014 KIT IWSLT Speech-to-Text Systems for English, German and Italian,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  19. I. Slawik, M. Mediani, J. Niehues, Y. Zhang, E. Cho, T. Herrmann, T.-L. Ha, and A. Waibel, “The KIT Translation Systems for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  20. M. Morchid, S. Huet, and R. Dufour, “A Topic-based Approach for Post-processing Correction of Automatic Translations,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. Proceedings of the 11 th International Workshop on Spoken Language Tr… find
  21. N. Segal, H. Bonneau-Maynard, Q. K. Do, A. Allauzen, J.-L. Gauvain, L. Lamel, and F. Yvon, “LIMSI English-French Speech Translation System,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  22. A. Rousseau, L. Barrault, P. Deléglise, Y. Estève, H. Schwenk, S. Bennacef, A. Muscariello, and S. Vanni, “The LIUM English-to-French Spoken Language Translation System and the Vecsys/LIUM Automatic Speech Recognition System for Italian Language for IWSLT 2014,” in Proceedings of the 11th Internat… find
  23. A. B. Romdhane, S. Jamoussi, A. B. Hamadou, and K. Smaili, “Phrase-based Language Modelling for Statistical Machine Translation,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  24. P. Shen, X. Lu, X. Hu, N. Kanda, M. Saiko, and C. Hori, “The NICT ASR System for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  25. X. Wang, A. Finch, M. Utiyama, T. Watanabe, and E. Sumita, “The NICT Translation System for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  26. K. Sudoh, G. Neubig, K. Duh, and K. Hayashi, “NTT-NAIST Syntax-based SMT Systems for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  27. K. Wolk and K. Marasek, “Polish - English Speech Statistical Machine Translation Systems for the IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  28. J. Wuebker, S. Peitz, A. Guta, and H. Ney, “The RWTH Aachen Machine Translation Systems for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  29. P. Bell, P. Swietojanski, J. Driesen, M. Sinclair, F. McInnes, and S. Renals, “The UEDIN ASR Systems for the IWSLT 2014 Evaluation,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  30. A. Birch, M. Huck, N. Durrani, N. Bogoychev, and P. Koehn, “Edinburgh SLT and MT System Description for the IWSLT 2014 Evaluation,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  31. R. W. M. Ng, M. Doulaty, R. Doddipatla, W. Aziz, K. Shah, O. Saz, M. Hasan, G. Alharbi, L. Specia, and T. Hain, “The USFD SLT system for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  32. S. Wang, Y. Wang, J. Li, Y. Cui, and L. Dai, “The USTC Machine Translation System for IWSLT 2014,” in Proceedings of the 11th International Workshop on Spoken Language Translation (IWSLT), Lake Tahoe, CA, 2014. find
  33. P. Koehn, “Europarl: A parallel corpus for statistical machine translation,” in Proceedings of the Tenth Machine Translation Summit (MT Summit X), Phuket, Thailand, September 2005, pp. 79–86. find
  34. H. Tseng, P. Chang, G. Andrew, D. Jurafsky, and C. Manning, “A conditional random field word segmenter,” in Fourth SIGHAN Workshop on Chinese Language Processing, 2005. find
  35. M. Federico, A. Cattelan, and M. Trombetti, “Measuring user productivity in machine translation enhanced computer assisted translation,” in Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas (AMTA), 2012. [Online]. Available: http://www.mt-archive.info/AMT… ↗
  36. S. Green, J. Heer, and C. D. Manning, “The efficacy of human post-editing for language translation,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2013, pp. 439–448. find
  37. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A Study of Translation Edit Rate with Targeted Human Annotation,” in Proceedings of the The Seventh Conference of the Association for Machine Translation in the Americas (AMTA), Cambridge, USA, 2006, pp. 223–231. find our card
  38. E. W. Noreen, Computer Intensive Methods for Testing Hypotheses: An Introduction. Wiley Interscience, 1989. find
  39. N. Chinchor, L. Hirschman, and D. D. Lewis, “Evaluating message understanding systems: An analysis of the third message understanding conference (muc-3),” Computational Linguistics, vol. 19, no. 3, pp. 409–449, 1993. find
  40. S. Riezler and J. T. Maxwell, “On some pitfalls in automatic evaluation and significance testing for MT,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Proceedings of the 11 th International Workshop on Spoken Language Translation Lake Tahoe, December 4th and… find
  41. B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. Chapman and Hall, 1993. find