Elamur Research⌕
☰
home / papers / 2022 / End-to-End Speech Translation of Arabic to English Broadcast News

End-to-End Speech Translation of Arabic to English Broadcast News

Paper · 2022 · Fethi Bougares and Salim Jouili

Summary

Arabic-to-English speech translation had almost no dedicated training data. The only public resource, CoVoST2, offers about 6 hours for this pair. The authors noticed that part of the LDC GALE parallel text corpora originated as English translations of the human-made transcripts distributed with GALE's speech recognition data. They matched these transcripts exactly against the source side of the MT corpora and aligned the matches using ASR timestamps. This produced a three-way corpus of Arabic audio, Arabic transcript and English translation, roughly 92 hours of broadcast news and conversation, which they cleaned and split into train, dev and test sets.

They compared cascaded systems (ESPnet ASR followed by FAIRSEQ LSTM MT) with end-to-end systems under two conditions: constrained (only the three-way data) and unconstrained (all GALE ASR and MT data). Two techniques improved the end-to-end model:

  • initializing its encoder from a pretrained ASR model;
  • adding about 795 hours of synthetic speech translation data, made by machine-translating all GALE transcripts.

The cascade still scored best. The work matters because it delivers the first end-to-end broadcast-news ST results for Arabic and shows how to recover usable resources from releases that were published separately.

Key points

  • The extracted corpus covers 92h10min and 39,635 segments, taken from 951h32min of GALE Phase 2/3/4 audio. The final split is train 83h54 (32,099 sentences), dev 1188 segments and test 987 segments, drawn from channels such as Abu Dhabi TV, Al Arabiya, Aljazeera and Syria TV.
  • ASR is an ESPnet attention encoder-decoder with VGGBLSTM layers. MT uses the FAIRSEQ lstm_luong_wmt_en_de template. Evaluation uses case-sensitive BLEU (mteval-v14) and WER.
  • ASR WER on dev fell from 20.90 (constrained) to 13.10 when trained on all GALE data, with 14.60 on test for the unconstrained model.
  • The synthetic corpus has 795 hours and 314,167 sentences. Adding it lifted end-to-end ST to 23.23 BLEU on dev and 18.97 on test. Combining it with ASR encoder initialization gave 24.95 and 19.09.
  • The cascade remained ahead overall, which the authors attribute to having only about 84 hours of genuine ST training data.

Interesting ideas

  • I1. Two resources released independently can be joined into an aligned multi-layer corpus by requiring exact string matches on a shared layer, here the transcript text; this yields aligned data cheaply and with high precision.
  • I2. A strong text-only model can label a large pool of unpaired inputs, and the resulting synthetic pairs can train the harder end-to-end model alongside a small set of real pairs.
  • I3. Results are reported under a constrained setting (only the matched data) and an unconstrained setting (all auxiliary data), so readers can tell how much of a gain comes from extra resources rather than the method itself.
  • I4. Instead of relying only on aggregate BLEU, the authors picked cases where an early-stage error spread into the final cascade output and checked whether the direct model avoided it.
  • I5. References made from text rather than from the original speech can penalize correct outputs that are phrased differently, so a metric may be biased by how its references were produced.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treebanks, VerbNet/FrameNet example sentences and QA sets that happen to share identical sentences could be joined this way, giving silver records that carry tree, frame and answer layers together.
  • I2 (learn): The LLM teacher tiers or agreeing witness parsers could label raw text in bulk, and the student parser could be trained on a mix of that silver data and gold data, tracking how much each portion helps.
  • I3 (learn): Evaluations could routinely report a gold-only run next to a gold-plus-silver run, which works like a negative control on whether silver data is really responsible for an improvement.
  • I4 (answer): When a fable QA answer is wrong, the system could trace whether the fault started in tagging, parsing or frames, because the explainable pipeline makes stage-level error attribution possible.
  • I5 (back): When scoring text → graph → text generation or translation against references, paraphrases that keep the meaning should not be penalized. Comparing meaning graphs rather than surface overlap would avoid this bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Le Mans UniversitéLe Mans · France16 works · 31,097 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksSpeech & spoken language · 519 works

Cite

Cite as Fethi Bougares and Salim Jouili. 2022. End-to-End Speech Translation of Arabic to English Broadcast News. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.29

@inproceedings{bougares2022end,
  title = {{End-to-End Speech Translation of Arabic to English Broadcast News}},
  author = {Fethi Bougares and Salim Jouili},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.29},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.29}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander Waibel, and Changhan Wang. 2020. find
  2. FINDINGS OF THE IWSLT 2020 EVALUA-TION CAMPAIGN. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 1–34, Online. Association for Computational Linguistics. find
  3. Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. CoRR, abs/1809.01431. ↗
  4. J. Cohen. 2007. The gale project: A description and an update. In 2007 IEEE Workshop on Automatic Speech Recognition Understanding (ASRU), pages 237–237. find
  5. Fahim Dalvi, Yifan Zhang, Sameer Khurana, Nadir Durrani, Hassan Sajjad, Ahmed Abdelali, Hamdy Mubarak, Ahmed Ali, and Stephan Vogel. 2017. QCRI live speech translation system. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 61–64, Valencia, Spain. Association for Computational Linguistics. find
  6. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012–2017, Minneapolis, Minnesota. ACL.
  7. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchís, Jorge Civera, and Alfons Juan. 2019. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In CoRR, volume abs/1911.03167. ↗
  8. Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting librispeech with french translations: A multimodal corpus for direct speech translation evaluation. In CoRR, volume abs/1802.03142. (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Michael Paul. 2006. Overview of the IWSLT 2006 evaluation campaign. In 2006 International Workshop on Spoken Language Translation, IWSLT 2006, Keihanna Science City, Kyoto, Japan, November 27-28, 2006, pages 1–15. ISCA. find
  10. David Stallard, Rohit Prasad, Prem Natarajan, Fred Choi, Shirin Saleem, Ralf Meermeier, Kriste Krstovski, Shankar Ananthakrishnan, and Jacob Devlin. 2011. The bbn transtalk speech-to-speech translation system. In Ivo Ipsic, editor, Speech and Language Technologies, chapter 3. IntechOpen, Rijeka. ↗
  11. Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020a. Covost: A diverse multilingual speech-to-text translation corpus. In CoRR. ↗
  12. Changhan Wang, Anne Wu, and Juan Pino. 2020b. Covost 2 and massively multilingual speech-to-text translation. In CoRR. ↗
  13. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018a. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207–2211. find
  14. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018b. Espnet: End-to-end speech processing toolkit. CoRR, abs/1804.00015. ↗
  15. Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. In CoRR, volume abs/1904.08075. ↗
  16. Brian Yan, Patrick Fernandes, Siddharth Dalmia, Jiatong Shi, Yifan Peng, Dan Berrebbi, Xinyi Wang, Graham Neubig, and Shinji Watanabe. 2022. find
  17. CMU’s IWSLT 2022 dialect speech translation system. In Proceedings of the 19th International Conference on Spoken Language Translation, pages 298–307, Dublin, Ireland (in-person and online). Association for Computational Linguistics. find
  18. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. ACL. ↗
  19. Jinyi Yang, Amir Hussein, Matthew Wiesner, and Sanjeev Khudanpur. 2022. find
  20. JHU IWSLT 2022 dialect speech translation system description. In Proceedings of the 19th International Conference on Spoken Language Translation, pages 319–326, Dublin, Ireland (in-person and online). ACL. find
  21. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019a. fairseq: A fast, extensible toolkit for sequence modeling. ↗
  22. Marcely Zanon Boito, John Ortega, Hugo Riguidel, Antoine Laurent, Loïc Barrault, Fethi Bougares, Firas Chaabani, Ha Nguyen, Florentin Barbier, Souhir Gahbiche, and Yannick Estève. 2022. ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks. In Proceedings of the 19th International Conference on Spoken Language Translation, pages 308–318, Dublin, Ireland (in-person and online). Association for Computational Linguistics. ↗
  23. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019b. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bougares-2022-end-end-speech-translation-arabic/