Elamur Research⌕
☰
home / papers / 2020 / SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation

SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation

Summary

Simultaneous translation systems start producing output before they have seen the whole source text or audio. They must therefore be judged on two things: translation quality and latency, meaning how much of the input was consumed before each output piece appeared. Before this work, papers computed latency inconsistently. Segmentation of text and speech was handled differently, and decoding and measurement code was rarely shared, so results could not be compared. The authors present SimulEval, an open-source toolkit for evaluating both text and speech simultaneous translation. It uses a server-client design:

  • The server holds the source and reference data, hands out input segments on request, logs the delay of every emitted output segment, and computes the scores.
  • The client runs a user-written agent that chooses whether to read more input or write output.

The toolkit reports standard quality and latency metrics. It also redefines the text latency metrics for speech, including a fix to Average Lagging so that systems which stop translating too early are not rewarded. A web viewer shows the decoding process. SimulEval was used to score the IWSLT 2020 simultaneous speech translation shared task.

Key points

  • Supported metrics:
    • Quality: BLEU, TER and METEOR.
    • Latency: Average Proportion (AP), Average Lagging (AL) and Differentiable Average Lagging (DAL).
    • Users can also plug in their own metrics.
  • How delay is measured: for text, it is the number of source words read. For speech, it is the total duration of the audio segments consumed so far. The speech versions of AP, AL and DAL are rewritten in these duration terms.
  • Why AP is problematic: AP depends on sequence length. For a wait-3 model it gives 0.72 when source and target have 10 tokens, but 0.52 when they have 100. Its values are also unevenly spread over the [0, 1] range.
  • Client and server setup: the two communicate over RESTful APIs and can run together or separately.
    • Users subclass an agent and implement policy (the read/write decision) and predict (the next output token).
    • Optional pre-processing (e.g., filterbank features, subword splitting) and post-processing (e.g., detokenization) can be added.
    • Interrupted evaluations can be resumed.
  • Use at IWSLT 2020: an earlier version scored all submissions to the shared task. Participants sent in Docker images containing their agents, and the test set, including its source side, was never released to them.

Interesting ideas

  • I1. Hiding the test set behind a server. Participants submit an agent and only ever receive input segments one at a time. This keeps the test data private and forces identical evaluation conditions regardless of each team's framework. The same pattern could suit other shared tasks with streaming or incremental input.
  • I2. Benchmarking against the reference, not the hypothesis. In the speech version of AL, the ideal policy is built from the reference length rather than the system's own output length. Otherwise, a system that ends its output prematurely could look deceptively fast. This is a general lesson: a metric's baseline should not depend on what the evaluated system itself produced.
  • I3. Testing metrics with toy cases. The authors check simple properties such as length invariance and even spread of values, using a fixed wait-k policy at different lengths. Running controlled examples like this is a cheap way to vet any new evaluation metric before adopting it.
  • I4. Separating decision logic from bookkeeping. The agent only contains the read/write policy and prediction. A separate state object handles preprocessing, context memory and postprocessing. Users can therefore focus on the policy, and the same evaluation harness serves both text and speech.
  • I5. Visualizing the decoding timeline. An interactive view lets users match source positions with the output emitted at that point. This makes latency behaviour easy to inspect and debug, and could carry over to any incremental model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-demos.19

@inproceedings{ma2020simuleval,
  title = {{SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation}},
  author = {Ma, Xutai and Dousti, Mohammad Javad and Wang, Changhan and Gu, Jiatao and Pino, Juan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.19},
  url = {https://doi.org/10.18653/v1/2020.emnlp-demos.19}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. 2018. Prediction improves simultaneous neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3022–3027. find
  2. Ebrahim Ansari, amittai axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander Waibel, and Changhan Wang. 2020. find
  3. FINDINGS OF THE IWSLT 2020 EVALU-ATION CAMPAIGN. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 1–34, Online. Association for Computational Linguistics. find
  4. Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. 2019. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1313–1323, Florence, Italy. Association for Computational Linguistics. find
  5. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  6. Colin Cherry and George Foster. 2019. Thinking slow about latency evaluation for simultaneous machine translation. arXiv preprint arXiv:1906.00048. ↗
  7. Kyunghyun Cho and Masha Esipova. 2016. Can neural machine translation do simultaneous translation? arXiv preprint arXiv:1606.02012. ↗
  8. Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. 2014. Don’t until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP), pages 1342–1352. find
  9. Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. 2017. Learning to translate in real-time with neural machine translation. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, pages 1053–1062. Association for Computational Linguistics (ACL). find
  10. Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3025–3036, Florence, Italy. Association for Computational Linguistics. ↗ our card
  11. Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020. Monotonic multihead attention. In International Conference on Learning Representations. ↗
  12. Dirk Merkel. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  14. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  15. Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao QIN, Zhou Zhao, and Tie-Yan Liu. 2020. SimulSpeech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787–3796, Online. Association for Computational Linguistics. find
  16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. Cambridge, MA. find our card
  17. Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. 2019. Simultaneous translation with flexible policy via restricted imitation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5816–5822, Florence, Italy. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2020-simuleval-evaluation-toolkit-simultaneous-translation/