Elamur Research⌕
☰
home / papers / 2022 / Representing 'how you say' with 'what you say': English corpus of focused speech and text reflecting corresponding implications

Representing 'how you say' with 'what you say': English corpus of focused speech and text reflecting corresponding implications

Paper · 2022 · Naoaki Suzuki and Satoshi Nakamura

Summary

Speech translation pipelines usually work from an ASR transcript. Two utterances that differ only in which word carries sentence stress therefore get identical translations, and the implied contrast disappears. The authors focus on one kind of emphasis, focus, which signals that alternatives exist: stressing "John" versus stressing "bought" changes what is being contrasted. They propose an intermediate step. Prosodic focus is first rewritten as English text that marks the same focus through words or constructions, such as focus particles, reflexives, clefts, pseudo-clefts, inversion or passives. That text is then translated.

To make such a model trainable, they built a corpus. Short Flickr8k captions were recorded by UK crowdworkers through a custom MTurk web recorder, with stress placed on each word in turn, including closed-class words. Each reading is paired with written text expressing its implication. Their analysis shows that the mapping from stress to wording draws on many different lexical and grammatical strategies. No earlier English dataset combined focus-varied speech with implication-reflecting text, so this resource fills a gap. It is to be released through LDC.

Key points

  • Captions from Flickr8k were filtered with several rules: only a final period as punctuation, left unchanged by GECToR, not a bare noun phrase, at most six words, no duplicates, and one caption per image. This left 1375 sentences, and the first 196 became the source set.
  • The six-word cap comes from the London-Lund corpus, where six is the third quartile of words per intonation phrase. This keeps each sentence a single IP with one sentence stress.
  • Each caption was targeted for recording by three different speakers. They recorded every focus position plus a neutral reading, and could skip any stress placement that felt unnatural.
  • The web recorder ran Google speech-to-text as a live check and showed "Speak Clearly" when no English was detected. It also captured two extra seconds of silence after each take to record ambient noise.
  • Earlier related corpora either varied only the intensity of emphasis on a fixed adjective, or collected focus-varied speech without its implied meaning.

Interesting ideas

  • I1. Before translating, the authors convert a non-textual signal (prosodic focus) into explicit text in the same language, so later stages only ever see words.
  • I2. The paper lists the devices English uses to express focus: particles (only, even, alone, let alone, particularly), reflexive intensifiers, it-clefts, pseudo-clefts, inversion and passivization. Many of these shift the focused item toward the end of the sentence.
  • I3. The authors set a sentence-length limit from corpus statistics (a quartile of IP length) rather than choosing it arbitrarily, so a confound stays controlled.
  • I4. Crowd input passed an automatic gate (an ASR check) before acceptance, and annotators could openly skip items that felt unnatural instead of forcing output.
  • I5. The corpus is built from minimal pairs: identical words, different focus, different implied meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could represent focus as an explicit alternative set with an exclusion implicature, whether it comes from stress annotation, "only", or a cleft, so every source yields one explainable frame.
  • I2 (tree): rules over UD trees could detect clefts (expl "it" + copula + relative clause) and pseudo-clefts and tag the focused constituent for prag downstream.
  • I3 (learn): caps on sentence length or complexity for silver data and evaluation sets could be derived from treebank distributions in the same way, with the reason documented.
  • I4 (learn): LLM teacher tiers and witness parsers could be allowed to abstain, with abstentions logged as a signal of unnatural or ambiguous inputs rather than treated as noise.
  • I5 (answer): such pairs could serve as negative controls, checking that an answer changes when focus changes (e.g. "who bought it?" versus "what did John do?") and stays put when only irrelevant prosody differs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nara Institute of Science and TechnologyIkoma · Japan34 works · 4,419 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Naoaki Suzuki and Satoshi Nakamura. 2022. Representing 'how you say' with 'what you say': English corpus of focused speech and text reflecting corresponding implications. arXiv:2203.15483.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.15483 · arXiv 2203.15483

@misc{suzuki2022representing,
  title = {{Representing 'how you say' with 'what you say': English corpus of focused speech and text reflecting corresponding implications}},
  author = {Naoaki Suzuki and Satoshi Nakamura},
  year = {2022},
  doi = {10.48550/arxiv.2203.15483},
  eprint = {2203.15483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.15483}
}

References

38 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. V. Mitra, S. Booker, E. Marchi, D. S. Farrar, U. D. Peitz, B. Cheng, E. Teves, A. Mehta, and D. Naik, “Leveraging acoustic cues and paralinguistic embeddings to detect expression from voice,” arXiv preprint arXiv:1907.00112, 2019. ↗
  2. K. J. Kohler, “What is emphasis and how is it coded,” in Proc. of Speech Prosody, 2006, pp. 748–751. find
  3. M. E. Rooth, “Association with focus,” Ph.D. dissertation, University of Massachusetts Amherst, 1985. find
  4. M. Rooth, “A theory of focus interpretation,” Natural Language Semantics, vol. 1, no. 1, pp. 75–116, 1992. ↗
  5. M. Krifka, “Basic notions of information structure,” Acta Linguistica Hungarica, vol. 55, no. 3-4, pp. 243–276, 2008. ↗
  6. J. C. Wells, English intonation: An introduction. Cambridge University Press, 2006. find
  7. T. M. Derwing, R. I. Thomson, J. A. Foote, and M. J. Munro, “A longitudinal study of listening perception in adult learners of English: Implications for teachers,” Canadian Modern Language Review, vol. 68, no. 3, pp. 247–266, 2012. ↗
  8. Y. Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y. Wu, “Direct speech-to-speech translation with a sequence-to-sequence model,” arXiv preprint arXiv:1904.06037, 2019. ↗
  9. P. Aguero, J. Adell, and A. Bonafonte, “Prosody generation for speech-to-speech translation,” in Proc. IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), vol. 1. IEEE, 2006, pp. 557–560. find
  10. M. Akagi, X. Han, R. Elbarougy, Y. Hamada, and J. Li, “Emotional speech recognition and synthesis in multiple languages toward affective speech-to-speech translation system,” in Proc. 10th International Conference on Intelligent Information Hiding and Multimedia Signal Processing. IEEE, 2014, pp. 574–577. find
  11. T. Kano, S. Sakti, S. Takamichi, G. Neubig, T. Toda, and S. Nakamura, “A method for translation of paralinguistic information,” in Proc. International Workshop on Spoken Language Translation (IWSLT), 2012. find
  12. T. Kano, S. Takamichi, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Generalizing continuous-space translation of paralinguistic information.” in Proc. INTERSPEECH, vol. 445, 2013, pp. 25– 29. find
  13. G. K. Anumanchipalli, L. C. Oliveira, and A. W. Black, “Intent transfer in speech-to-speech machine translation,” in Proc. IEEE Spoken Language Technology Workshop (SLT), 2012, pp. 153– 158. find
  14. A. Tsiartas, P. G. Georgiou, and S. S. Narayanan, “Toward transfer of acoustic cues of emphasis across languages.” in Proc. INTER-SPEECH, 2013, pp. 3483–3486. find
  15. Q. T. Do, S. Sakti, G. Neubig, and S. Nakamura, “Transferring emphasis in speech translation using hard-attentional neural network models.” in Proc. INTERSPEECH, 2016, pp. 2533–2537. find
  16. Q. T. Do, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Preserving word-level emphasis in speech-to-speech translation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 3, pp. 544–556, 2016. find
  17. Q. T. Do, S. Sakti, and S. Nakamura, “Sequence-to-sequence models for emphasis speech translation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 10, pp. 1873–1883, 2018. find
  18. A. Cruttenden, Intonation. Cambridge University Press, 1997. find
  19. L. J. Downing, “The prosody and syntax of focus in chitumbuka,” ZAS Papers in Linguistics, vol. 43, pp. 55–79, 2006. find
  20. K. Hartmann, “Focus and tone,” Acta Linguistica Hungarica, vol. 55, no. 3-4, pp. 415–426, 2008. ↗
  21. Q. T. Do, S. Sakti, and S. Nakamura, “Toward multi-features emphasis speech translation: Assessment of human emphasis production and perception with speech and text clues,” in Proc. 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 700–706. find
  22. H. Tokuyama, S. Sakti, K. Sudoh, and S. Nakamura, “Transcribing paralinguistic acoustic cues to target language text in transformer-based speech-to-text translation,” Proc. Interspeech 2021, pp. 2262–2266, 2021. find
  23. S. Latif, I. Kim, I. Calapodescu, and L. Besacier, “Controlling prosody in end-to-end TTS: A case study on contrastive focus generation,” in Proc. 25th Conference on Computational Natural Language Learning (CoNLL), 2021, pp. 544–551.
  24. S. Greenbaum, A student’s grammar of the English language. Pearson Education India, 1990. find
  25. C. Boonthum, “iSTART: Paraphrase recognition,” in Proc. ACL Student Research Workshop, 2004, pp. 31–36. find
  26. W. Dolan, C. Quirk, C. Brockett, and B. Dolan, “Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources,” in Proc. 20th International Conference on Computational Linguistics (ACL), 2004.
  27. M. Vila, M. A. Martı́, H. Rodrı́guez et al., “Is this a paraphrase? What kind? Paraphrase boundaries and typology,” Open Journal of Modern Linguistics, vol. 4, no. 01, p. 205, 2014. ↗
  28. J. Taglicht, Message and emphasis: On focus and scope in English. Addison-Wesley Longman Limited, 1984, no. 15. find
  29. E. König, The meaning of focus particles: A comparative perspective. Routledge, 2002. find
  30. C. J. Fillmore, P. Kay, and M. C. O’connor, “Regularity and idiomaticity in grammatical constructions: The case of let alone,” Language, pp. 501–538, 1988. ↗
  31. E. König and V. Gast, “Focused assertion of identity: A typology of intensifiers,” Linguistic Typology, vol. 10, pp. 223–76, 2006. ↗
  32. B. Aarts, Oxford modern English grammar. Press, 2011. Oxford University find
  33. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, “Collecting image annotations using amazon’s mechanical turk,” in Proc. NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, 2010, pp. 139–147.
  34. K. Omelianchuk, V. Atrasevych, A. Chernodub, and O. Skurzhanskyi, “GECToR–grammatical error correction: Tag, not rewrite,” in 15th Workshop on Innovative Use of NLP for Building Educational Applications, 2020, pp. 163–170. find
  35. J. Svartvik, The London–Lund corpus of spoken English: Description and research. Lund University Press, 1990, vol. 82. find
  36. R. Kennedy, S. Clifford, T. Burleigh, P. D. Waggoner, R. Jewell, and N. J. Winter, “The shape of and solutions to the MTurk quality crisis,” Political Science Research and Methods, vol. 8, no. 4, pp. 614–629, 2020. ↗
  37. M. E. Whiting, G. Hugh, and M. S. Bernstein, “Fair work: Crowd work minimum wage with one line of code,” in Proc. AAAI Conference on Human Computation and Crowdsourcing, vol. 7, 2019, pp. 197–206. find
  38. R. Barzilay and K. McKeown, “Extracting paraphrases from a parallel corpus,” in Proc. 39th annual meeting of the Association for Computational Linguistics, 2001, pp. 50–57.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suzuki-2022-representing-say-say-english-corpus/