Elamur Research⌕
☰
home / papers / 2026 / S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature

S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature

Summary

Synthetic audiobook narration still struggles to voice fictional characters convincingly. Assigning each character a fitting voice requires knowing traits like age, gender or origin, which novels often leave implicit or spread across distant chapters. Earlier character datasets mostly offer free-form literary descriptions rather than voice-relevant traits. The authors introduce S-VoCAL, a benchmark of 952 character-book pairs from 192 Project Gutenberg novels. Each pair is labeled with eight attributes motivated by sociophonetic research. Labels were obtained by linking characters to Wikidata, and Age was further completed through manual annotation. The evaluation framework scores each attribute differently. It uses strict and soft F1 with weighting for closed or semi-closed classes. For open-text attributes, it uses a similarity measure built on instruction-aware Qwen-3 8b embeddings, which matches human judgments better than BERTScore. A baseline RAG pipeline retrieves passages with E5-large and lets LLMs infer the attributes. It handles Age, Gender and Spoken languages well but falters on Origin, Occupation and Physical Health. The resource gives TTS and NLP work a targeted test for voice-oriented character profiling.

Key points

  • Dataset composition: S-VoCAL covers 952 character-book pairs across 192 unique novels from Project Gutenberg.
  • Attributes: The eight attributes are Age, Gender, Origin, Residence, Occupation, Spoken languages, Physical health and Type (human vs. non-human). Name and Aliases are also included.
  • Coverage varies widely:
    • Gender is available for 96.5% of characters.
    • Physical health is available for only 4.1%.
    • Age coverage rose from 4.1% to 37.7% after manual annotation.
  • Baseline setup: Passages are retrieved with E5-large using attribute-specific query templates. Qwen3 and Phi-4 then infer the attributes. Results are compared with human-aligned scores (mHAS).
  • Open-class similarity scores (Qwen3 / Phi-4 / baseline):
    • Type (non-human): 0.869 / 0.810 / 0.569.
    • Physical Health: about 0.30–0.33 for all three.
    • The two LLMs tend to make errors on different instances.
  • Availability: Data and evaluation code are released at github.com/AbigailBerthe/S-VoCAL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4hnrkbo6dhi5

@inproceedings{berthepardo2026s,
  title = {{S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature}},
  author = {Berthe-Pardo, Abigail and Michel, Gaspard and Epure, Elena V. and Cerisara, Christophe},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4hnrkbo6dhi5},
  url = {https://doi.org/10.63317/4hnrkbo6dhi5}
}

References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Sali… find our card
  2. David Abercrombie. 1967. Elements of General Phonetics. Edinburgh University Press, Edinburgh. find
  3. David Bamman, Ted Underwood, and Noah A Smith. 2014. A bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370–379. find our card
  4. Sabyasachee Baruah and Shrikanth Narayanan. 2024. Character attribute extraction from movie scripts using llms. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8270– 8275. find
  5. Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. " let your characters tell their story": A dataset for character-centric narrative understanding. arXiv preprint arXiv:2109.05438. find
  6. James Browm. 2021. The Art of Storytelling – How Voice Over Brings Audiobooks to Life. https://jamesbrownvoice.com/voice-over-bringing-audiobooks-to-life. ↗
  7. Gerald J. Canter. 1963a. Speech characteristics of patients with parkinson’s disease. i. intensity, pitch and duration. Journal of Speech and Hearing Disorders, 28:221–229. find
  8. Gerald J. Canter. 1963b. Speech characteristics of patients with parkinson’s disease: I. intensity, pitch, and duration. Journal of Speech and Hearing Disorders, 28(3):221–229. find
  9. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeology of books known to ChatGPT/GPT-4. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312–7327, Singapore. Association for Computational Linguistics. find our card
  10. Ella Creamer. 2025. UK audiobook revenue up by almost a third last year . https://shorturl.at/5IRYk. ↗
  11. Alan Cruttenden. 1986. Intonation. Cambridge University Press, Cambridge. find
  12. Albert Di Cristo. 2013. La prosodie de la parole. De Boeck Superieur. find
  13. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  14. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. find
  15. Sascha Frühholz and Pascal Belin. 2018. The Oxford Handbook of Voice Perception. Oxford University Press. find
  16. Pablo Gervás and Gonzalo Méndez. 2024. Tagging narrative with propp’s character functions using large language models. In Text2Story@ ECIR, pages 137–148. find
  17. Brittany Goodwin. 2024. So You Want To Narrate Audiobooks. https://www.brittanygoodwin.com/blog/2019/12/8/ so-you-want-to-narrate-audiobooks. ↗
  18. Alexander Gurung and Mirella Lapata. 2024. Chiron: Rich character representations in long-form narratives. In Findings of the Association for Computational Linguistics: EMNLP 2024, page 8523–8547. Association for Computational Linguistics. find
  19. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings — narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943, Miami, Florida, USA. Association for Computational Linguistics. find
  20. Harry Hollien and Thomas Shipp. 1972a. Speaking fundamental frequency and chronologic age in males. Journal of Speech and Hearing Research, 15(1):155–159. find
  21. Harry Hollien and Thomas F. Shipp. 1972b. Speaking fundamental frequency and chronological age in males. Journal of Speech and Hearing Research, 15:155–159. find
  22. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? find our card
  23. Elias Iosif and Taniya Mishra. 2014. From speaker identification to affective analysis: A multi-step system for analyzing children’s stories. In Proceedings of the 3rd Workshop on Computational Linguistics for Literature (CLFL), pages 40–49, Gothenburg, Sweden. Association for Computational Linguist… find
  24. Brandon Jaipersaud, Zining Zhu, Frank Rudzicz, and Elliot Creager. 2024. Show, don’t tell: Uncovering implicit character portrayal using llms. find
  25. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  26. William Labov. 1973. Sociolinguistic patterns. 4. University of Pennsylvania press. find
  27. John Laver and Peter Trudgill. 1979. Phonetic and linguistic markers in speech. Social markers in speech, 1:32. find
  28. John Laver and Peter Trudgill. 1991. Phonetic and linguistic markers in speech. In John Laver, editor, The Gift of Speech, pages 235–264. Edinburgh University Press, Edinburgh. find
  29. Louis Mahon and Mirella Lapata. 2024. A modular approach for multimodal summarization of tv shows. find
  30. Gaspard Michel, Elena V. Epure, and Christophe Cerisara. 2025. Libriquote: A speech dataset of fictional character utterances for expressive zero-shot speech synthesis. find
  31. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L… find our card
  32. Francis Nolan. 1987. Alan cruttenden, intonation. cambridge: Cambridge university press, 1986. pp. xiv + 214. Journal of Linguistics, 23(2):455–458. find
  33. Argyrios Papoudakis, Mirella Lapata, and Frank Keller. 2024. BookWorm: A dataset for character description and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4471–4500, Miami, Florida, USA. Association for Computational Linguistics. find
  34. Kyeongman Park, Seongho Joo, and Kyomin Jung. 2025. Multiactor-audiobook: Zero-shot audiobook generation with faces and voices of multiple speakers. find
  35. Herman Parret. 2002. Chapitre 2. La qualité de la voix, Le Point philosophique, pages 39–66. De Boeck Supérieur, Louvain-la-Neuve. find
  36. Charuta Pethe, Bach Pham, Felix D Childress, Yunting Yin, and Steven Skiena. 2025. Prosody analysis of audiobooks. In 2025 19th International Conference on Semantic Computing (ICSC), page 217–221. IEEE. find
  37. Emma Rodero and Ignacio Lucas. 2023. Synthetic versus human voices in audiobooks: The human emotional intimacy effect. New Media & Society, 25(7):1746–1764. find
  38. Nilanjana Roy. 2024. Do audiobooks count as reading? Financial Times. find
  39. Piotr Sorokowski, Katarzyna Pisanski, Tomasz Frąckowiak, Aleksander Kobylarek, and Agata Groyecka-Bernard. 2024. Voice-based judgments of sex, height, weight, attractiveness, health, and psychological traits based on free speech versus scripted speech. Psychonomic Bulletin & Review, pages 1–10. find
  40. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and gpt-3: Automated extraction of character roles without training data. find
  41. Statista. 2025a. Audiobooks - China. https://www.statista.com/outlook/amo/media/books/audiobooks/china. ↗
  42. Statista. 2025b. Audiobooks - Europe. https://www.statista.com/outlook/amo/media/books/audiobooks/europe. ↗
  43. Statista. 2025c. Audiobooks - United States. https://www.statista.com/outlook/amo/media/books/audiobooks/united-states. ↗
  44. Kobie Van Krieken, Hans Hoeken, and José Sanders. 2017. Evoking and measuring identification with narrative characters–a linguistic cues framework. Frontiers in psychology, 8:1190. find
  45. Krishnapriya Vishnubhotla, Adam Hammond, Graeme Hirst, and Saif Mohammad. 2024. The emotion dynamics of literary novels. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2557–2574, Bangkok, Thailand. Association for Computational Linguistics. find
  46. Brendan Walsh, Mark Hamilton, Greg Newby, Xi Wang, Serena Ruan, Sheng Zhao, Lei He, Shaofei Zhang, Eric Dettinger, William T. Freeman, and Markus Weimer. 2023. Large-scale automatic audiobook creation. find
  47. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024a. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. find
  48. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024b. Text embeddings by weakly-supervised contrastive pre-training. find our card
  49. Alissa P Wolters, Young-Suk Grace Kim, and John William Szura. 2022. Is reading prosody related to reading comprehension? a meta-analysis. Scientific Studies of Reading, 26(1):1– 20. find
  50. Detai Xin, Sharath Adavanne, Federico Ang, Ashish Kulkarni, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Improving speech prosody of audiobook text-to-speech synthesis with acoustic and textual contexts. find
  51. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  52. Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. 2024. Evaluating character understanding of large language models via character profiling from fictional works. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processi… find
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675. find
  54. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pardo-2026-s-vocal-dataset-evaluation-framework/