Elamur Research⌕
☰
home / papers / 2024 / Creating an Aligned Corpus of Sound and Text: The Multimodal Corpus of Shakespeare and Milton

Creating an Aligned Corpus of Sound and Text: The Multimodal Corpus of Shakespeare and Milton

Paper · 2024 · Manex Agirrezabal

Summary

The paper addresses a gap in poetry resources. Existing poetry corpora contain only text, so there is no way to check how metrical expectations are actually realized in speech. To fill this gap, the author builds a multimodal corpus. It pairs Shakespeare's Sonnets and Milton's Paradise Lost, taken from Project Gutenberg, with public-domain LibriVox recordings. Each collection is read by a single narrator.

The corpus is built by an automatic pipeline:

  • verse lines are matched to audio with Dynamic Time Warping (Aeneas);
  • each word gets a phonemic form from the CMU dictionary, with a G2P fallback;
  • syllables are split by rule from the phonemes;
  • phone timings come from HTK forced alignment via FAVE and are projected up to syllables and words;
  • every line receives a stress pattern from a pretrained BiLSTM+CRF scansion model.

Everything is stored in TEI XML with time stamps, and a small website lets users play audio from any clicked line. A first descriptive analysis finds that word duration tracks word length well, but line duration barely tracks line length. The resource opens empirical study of how stress, rhyme and meter surface acoustically.

Key points

  • Size: roughly 12.5 hours of audio, almost 100,000 tokens and about 17,000 types. Readers are Chris Hughes (Sonnets) and Thomas A. Copeland (Paradise Lost).
  • Aligned levels are line, word, syllable and phone. Each line in the TEI file has beginning/end times and a scansion string such as −+−+−+−+−+.
  • The scansion model builds on Lample et al.'s tagger and is trained on For Better For Verse. It scores 90.80% accuracy per syllable but only 53.29% per full line.
  • Pearson correlations between duration and length:
    • line seconds vs. characters: 0.2932
    • line seconds vs. words: 0.2514
    • word seconds vs. characters: 0.7163
  • Two error findings:
    • The rule-based syllabifier (Foma, adapted from Hulden 2006) is never more than one syllable off.
    • The scansion model fails on unknown words, emitting a single unstressed syllable even for long ones.

Interesting ideas

  • I1. Align coarse-to-fine: anchor long recordings at the line level first, then run fine forced alignment only inside each short segment, so errors cannot drift across the whole file.
  • I2. Use a lexicon first and a learned fallback second: look a word up in a curated dictionary (CMU) and call a G2P model only for misses.
  • I3. Compare two independent tools on a shared quantity: syllable counts from the syllabifier and from the scansion model are checked against each other. This exposed that unknown words collapse to one unstressed syllable.
  • I4. Report a simple sanity statistic against an intuitive expectation: the author expected positive duration–length correlation everywhere and found it strong only for words. This shows where naive assumptions break.
  • I5. Context-dependent prominence: the scansion model stresses the same function word ("with") differently within one line, depending on its surroundings.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data from long documents, first fix sentence or paragraph boundaries, then run detailed parsing and agreement checks only within those anchored spans.
  • I2 (tokens): The lemmatizer and morphology in crate en could follow the same order, with an explicit flag on fallback outputs so later steps know which analyses are less trusted.
  • I3 (learn): Mova can check cheap, tool-independent invariants (token counts, argument counts per frame) across witness parsers to find silent default behaviours for out-of-vocabulary input.
  • I4 (learn): Evaluation could include such expectation checks, e.g. a correlation between sentence length and tree depth, so that surprising deviations flag either a pipeline bug or a genuine property of the data.
  • I5 (tokens): The tagger could carry a contextual weak/strong-form attribute for function words, giving later explainable steps a hint about emphasis without relying on fixed per-word values.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Manex Agirrezabal. 2024. Creating an Aligned Corpus of Sound and Text: The Multimodal Corpus of Shakespeare and Milton. arXiv:2407.18730.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.18730 · arXiv 2407.18730

@misc{agirrezabal2024creating,
  title = {{Creating an Aligned Corpus of Sound and Text: The Multimodal Corpus of Shakespeare and Milton}},
  author = {Manex Agirrezabal},
  year = {2024},
  doi = {10.48550/arxiv.2407.18730},
  eprint = {2407.18730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.18730}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Delmonte, R. (2016). Exploring Shakespeare’s Sonnets with SPARSAR. Linguistics and Literature Studies, 4(1):61–95. Figure 7: The correlation between word duration in seconds and word length in characters. This shows a Pearson correlation of 0.7163. ↗
  2. Delmonte, R. and Bacalu, C. (2013). SPARSAR: a System for Poetry Automatic Rhythm and Style AnalyzeR. In Speech and Language Technology in Education. find
  3. González-Blanco García, E. and Rodríguez, J. L. (2013). ReMetCa: a TEI based digital repertory on Medieval Spanish poetry. find
  4. Hart, M. (1971). Project gutenberg. Project Gutenberg. find
  5. Hayes, B., Wilson, C., and Shisko, A. (2012). Maxent grammars for the metrics of Shakespeare and Milton. Language, 88(4):691–731. ↗
  6. Hulden, M. (2006). Finite-state syllabification. In Yli-Jyrä, A., Karttunen, L., and Karhumäki, J., editors, Finite-State Methods and Natural Language Processing, volume 4002 of Lecture Notes in Computer Science, pages 86–96. Springer. 00009. find
  7. Hulden, M. (2009). Foma: a finite-state compiler and library. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics: Demonstrations Session, pages 29–32. Association for Computational Linguistics. find
  8. Jacobs, A. M. (2018). The gutenberg english poetry corpus: exemplary quantitative narrative analyses. Frontiers in Digital Humanities, 5:5. ↗
  9. Kearns, J. (2014). LibriVox: Free Public Domain Audiobooks. Reference Reviews. Publisher: Emerald Group Publishing Limited. ↗
  10. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. In Proceedings of NAACL-2016, San Diego, California, USA. Association for Computational Linguistics. ↗
  11. MacKenzie, L. and Turton, D. (2020). Assessing the accuracy of existing forced alignment software on varieties of british english. Linguistics Vanguard, 6(s1):20180061. ↗
  12. Malta, M. C. and Gonzalez-Blanco, E. (2016). POSTDATA: towards publishing European poetry as linked open data. In Proceedings of the 2016 International Conference on Dublin Core and Metadata Applications, pages 19–20. Dublin Core Metadata Initiative. find
  13. Meyer-Sickendiek, B., Hussein, H., and Baumann, T. (2017). Rhythmicalizer: Data analysis for the identification of rhythmic patterns in readout poetry (work-in-progress). INFORMATIK, pages 2189–2200. find
  14. Meyer-Sickendiek, B., Hussein, H., and Baumann, T. (2018). Analysis and Classification of Prosodic Styles in Post-modern Spoken Poetry. INF-DH-2018. ↗
  15. Müller, M. (2007). Dynamic time warping. Information retrieval for music and motion, pages 69–84. find
  16. Navarro-Colorado, B., Lafoz, M. R., and Sánchez, N. (2016). Metrical Annotation of a Large Corpus of Spanish Sonnets: Representation, Scansion and Evaluation. In Proceedings of the Language Resources and Evaluation Conference. 00000. find
  17. Park, K. and Kim, J. (2019). g2pe. https://github.com/Kyubyong/g2p. ↗
  18. Pettarin, A. (2017). aeneas. find
  19. Pickering, W. (1832). The Poetical Works of John Milton, volume 2. William Pickering. 00000. find
  20. Plecháč, P. and Kolár, R. (2015). The Corpus of Czech Verse. Studia Metrica et Poetica, 2(1):107–118. ↗
  21. Rosenfelder, I., Fruehwald, J., Evanini, K., and Yuan, J. (2011). Fave (forced alignment and vowel extraction) program suite. URL http://fave.ling. upenn. edu. ↗
  22. Seuss (1953). Scrambled eggs super! Random House Books for Young Readers. 00005. find
  23. TEI Consortium (2008). TEI P5: Guidelines for electronic text encoding and interchange. TEI Consortium. ↗
  24. Tucker, H. F. (2011). Poetic data and the news from poems: A for better for verse memoir. Victorian Poetry, 49(2):267–281. ↗
  25. Weide, R. (1998). The CMU pronunciation dictionary, release 0.6. Carnegie Mellon University. find
  26. Yuan, J. and Liberman, M. (2008). Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agirrezabal-2024-creating-aligned-corpus-sound-text/