Overview of the IWSLT 2017 Evaluation Campaign

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2017.iwslt-1.1.pdf

Summary

This report describes the 2017 shared evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT). The organisers fixed the training data, test conditions and metrics ahead of time so that systems could be compared fairly. There were three tasks:

• Multilingual task: a single engine translates TED Talks among Dutch, English, German, Italian and Romanian. This includes zero-shot directions, where the language pair never appears together in the training data.
• Dialogue task: translating Japanese–English conversations, using context to handle pronouns and other omitted elements.
• Lecture task: recognising and translating spontaneous university lectures rather than polished talks.

Outputs were scored automatically with BLEU, NIST and TER through an evaluation server, and the multilingual task also received human assessment. The main finding is that zero-shot systems can come close to systems that saw the tested pair during training. Multilingual training also appears to reduce data sparsity almost as well as adding large out-of-domain corpora. The organisers also report a noticeable drop in participation and plan to discuss its causes with the community.

Key points

• Multilingual setup: 20 directions in total. Zero-shot pairs were Dutch↔German and Italian↔Romanian, trained only on the other 16 pairs. Pivoting and synthetic training data were not allowed. In-domain data came from WIT 3, averaging about 3.99M tokens and 160k sentences per pair, with tst2017 as the test set.
• Submissions: 7 teams took part overall (FBK, GTCT, KIT, KYOTO, RWTH, UEDIN, UDS-DFKI). The multilingual task received 9 primary runs from 5 sites.
• Zero-shot results (average BLEU): KYOTO's zero-shot system scored 20.55, against 20.60 for its small-data system. KIT reached 21.13 with large data and 20.97 with small data.
• Dialogue corpus: the development set has 1,476 utterances and the test set 1,510. Words were tokenised with KyTea for Japanese and the Moses tokenizer for English.
• Lecture task:
  • KIT's ASR achieved an overall WER of 8.5 on English and 22.8 on German.
  • For German→English translation, UEDIN scored 13.99 BLEU and KIT 12.50.
  • For English→German translation, KIT scored 18.59 BLEU and UEDIN 15.98.