📄 Findings of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23)

Summary

WMT-SLT23, the second edition of the WMT shared task on sign language translation, asks systems to translate between signed and spoken languages. This requires handling visual input such as video or pose estimates, so the problem goes beyond standard text-to-text MT. The organizers built on the 2022 edition in three ways: they grew from one language pair to three (DSGS, LSF-CH and LIS-CH paired with German, French and Italian), released far more DSGS training data, and placed more weight on spoken-to-signed translation by supplying official baselines for that direction.

Four teams took part, one of them building a baseline, and every team entered only the DSGS-to-German track. Both automatic and human evaluation showed very poor quality: the best system averaged about 1/100 in human scores, which is far from practical use. The authors attribute this to three gaps: too little usable training data, the shift in modality from text, and missing basic NLP tools for sign languages. The task still leaves useful resources behind, including new corpora, reproducible baselines, public system outputs and additional human judgments for SLT.

Key points

• The task had four tracks: DSGS→German, German→DSGS, French→LSF-CH and Italian→LIS-CH. No team submitted to any track with a sign language as the target.
• DSGS training data rose to 437 hours from 16 the previous year. Unlike 2022, the data was released without manual correction.
• The SRF23 training corpus behaves like a noisy comparable corpus. Its German subtitles follow the broadcast audio rather than the delayed live interpretation, and no participant attempted to fix the alignment.
• TTIC and CASIA scored higher on the SRF test data than on Signsuisse. In contrast, human reference translations rated poorly on SRF, partly because of bad video cuts and missing discourse context.
• For future editions, the organizers propose shared tasks on more basic SLP problems, namely alignment, segmentation and parallel-corpus filtering. They also suggest moving away from interpreted news material.