Elamur Research⌕
☰
home / papers / 2014 / Translations of the Callhome Egyptian Arabic corpus for conversational speech translation

Translations of the Callhome Egyptian Arabic corpus for conversational speech translation

Paper · 2014 · Gaurav Kumar, Yuan Cao, Ryan Cotterell et al.

Summary

Research on translating conversational speech has been held back by the lack of in-domain data. For Egyptian Arabic in particular, no resource aligned speech, transcripts and English translations from the same domain. As a result, translation systems were usually trained on scripted or written text that differs sharply from spontaneous phone talk. The authors address this by adding English translations to the existing LDC Callhome Egyptian Arabic telephone corpus, its HUB5 evaluation set and its supplement. They stripped transcription markup and collected four independent translations per utterance through Amazon Mechanical Turk. Several safeguards against low-quality or machine-generated work were layered together. In total, 838 contributors produced 143,568 English translations. Agreement among translators, measured with leave-one-out BLEU, falls between roughly 36% and 40% depending on the partition. The authors also plan to release ASR lattices, oracle paths and 1-best output. The result is an aligned resource combining speech, transcripts, recognizer output and translations, which supports research on cascaded and integrated speech translation of dialectal, disfluent conversation.

Key points

  • The source data are LDC Callhome Egyptian Arabic sets:
    • ECA-96 (LDC97S45/LDC97T19) has 120 conversations split 80/20/20 into train/dev/eval, with about 30k utterances.
    • 97-eval-H5 (LDC2002S22/LDC2002T39) has 20 conversations.
    • The ECA supplement (LDC2002S37/LDC2002T38) has 20 conversations.
  • Transcripts were first written in a phonemic romanization and then mapped to Arabic script via the LDC99L22 lexicon, which has 51,202 entries. Non-MSA words were deliberately left unnormalized.
  • Each MTurk HIT showed ten consecutive segments in conversation order, and each utterance received four translations.
  • Quality control combined four checks:
    • comparing submissions against Google Translate by edit distance;
    • rendering the source as an image to block copy-paste;
    • inserting three gold-standard segments per translator;
    • checking self-reported metadata, plus manual spot checks.
  • Cross-fold inter-annotator BLEU ranges from 35.64% (dev) to 40.09% (train).

Interesting ideas

  • I1. Scoring each human reference against the remaining ones gives an agreement ceiling, which tells you how much variation is legitimate before judging any system.
  • I2. Hiding known-answer gold items inside ordinary work batches exposes careless or gaming annotators without them noticing.
  • I3. Flagging submissions suspiciously close to a known automatic system catches people who merely relay machine output instead of doing the task.
  • I4. Presenting utterances in their original order with alternating speakers lets annotators use dialogue context for ambiguous short turns.
  • I5. Markup was removed for annotation, but segment and channel IDs were kept so outputs could be mapped back, and markup-only lines were reinserted afterward.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Measure leave-one-out agreement among witness parsers on silver data, and use it as an upper bound and sanity baseline when evaluating Mova's own parser.
  • I2 (learn): Mix pre-verified gold sentences into LLM teacher batches, and downgrade a teacher tier whose answers on those items drift from gold.
  • I3 (learn): Check whether a teacher's or witness's trees nearly duplicate a single baseline parser, and treat such cases as non-independent votes.
  • I4 (frames): When collecting frame or coreference labels for dialogues, feed neighboring turns along with the target sentence so pronoun and role decisions rely on discourse context.
  • I5 (tokens): Strip disfluency and noise markers before tagging while keeping stable span IDs, so the cleaned analysis stays traceable to the raw input.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 worksDialogue systems · 624 worksSpeech & spoken language · 480 works

Cite

Cite as Gaurav Kumar, Yuan Cao, Ryan Cotterell, Chris Callison-Burch, Daniel Povey and Sanjeev Khudanpur. 2014. Translations of the Callhome Egyptian Arabic corpus for conversational speech translation. Proceedings of the 11th International Workshop on Spoken Language Translation: Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kumar2014translations,
  title = {{Translations of the Callhome Egyptian Arabic corpus for conversational speech translation}},
  author = {Gaurav Kumar, Yuan Cao, Ryan Cotterell, Chris Callison-Burch, Daniel Povey and Sanjeev Khudanpur},
  booktitle = {Proceedings of the 11th International Workshop on Spoken Language Translation: Papers},
  year = {2014},
  url = {https://aclanthology.org/2014.iwslt-papers.13/}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Bender Richard Zens, “The RWTH phrase-based statistical machine translation system,” in Proceedings of the International Workshop on Spoken Language Translation (IWSLT), 2005, pp. 155–162. find
  2. E. Matusov, S. Kanthak, and H. Ney, “Integrating speech recognition and machine translation: Where do we stand?,” in 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings, May 2006, vol. 5, pp. V–V. find
  3. Scott Novotney and Chris Callison-Burch, “Cheap, fast and good enough: Automatic speech recognition with non-expert transcription,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Stroudsburg, PA, USA, 2010, HLT ’10, p. 207215, Association for Computational Linguistics. find
  4. A Sorokin and D. Forsyth, “Utility data annotation with amazon mechanical turk,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08, June 2008, pp. 1–8. find
  5. Aniket Kittur, Ed H. Chi, and Bongwon Suh, “Crowdsourcing user studies with mechanical turk,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2008, CHI ’08, p. 453456, ACM. find
  6. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng, “Cheap and fastbut is it good?: Evaluating non-expert annotations for natural language tasks,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, Source Translation 1 Translation 2 Translation 3 Translation 4 you do n’t reply to the phone so you do n’t answer the phone then you do n’t answer the phone it seems because you do n’t answer the call then Source Translation 1 Translation 2 Translation 3 Translation 4 he feels hard for himself too he feel bad about himself he feels sorry for himself too i feel sorrow about his condition too Table 5. A sample of the translations obtained using the translation task described in section 4. The translations are lower-cased, tokenized and punctuation has been normalized. USA, 2008, EMNLP ’08, p. 254263, Association for Computational Linguistics. find
  7. Chris Callison-Burch, “Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1, Stroudsburg, PA, USA, 2009, EMNLP ’09, p. 286295, Association for Computational Linguistics.
  8. Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis, “Running experiments on amazon mechanical turk,” SSRN Scholarly Paper ID 1626226, Social Science Research Network, Rochester, NY, June 2010. ↗
  9. Omar F. Zaidan and Chris Callison-Burch, “Crowdsourcing translation: Professional quality from non-professionals,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, Stroudsburg, PA, USA, 2011, HLT ’11, p. 12201229, Association for Computational Linguistics.
  10. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur, “Improved speech-to-text translation with the fisher and callhome translated corpus of spanish-english speech,” in Proceedings of the International Workshop on Spoken Language Translation (IWSLT), 2013. Proceedings of the 11 th International Workshop on Spoken Language Translation Lake Tahoe, December 4th and 5th, 2014  QJ.Ó ñJ K@ AÓ ù ®J  . K àñ ®J ÊJË@ úΫ €ðX àAÒ» é‚ ® K éJ Ê« àAJ.ª’Ó find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumar-2014-translations-callhome-egyptian-arabic-corpus/