Emotions In Oral History Interviews: A Multimodal Approach to Holocaust Testimonies

Paper · 2026 · Nele Mantaj, Vaibhav Agarwal and Ines Matres

Summary

The authors ask whether the audio and video of Holocaust oral history interviews carry emotional information that written transcripts lose, such as pauses, voice changes and facial expressions. During a 10-day Digital Humanities Hackathon at the University of Helsinki (2025), a student team took 10 roughly hour-long USHMM interviews, five in English and five in Polish. They split each one into text, audio and video layers, processed each layer separately and then realigned them for side-by-side comparison.

Text: PDF transcripts were converted, cut into utterances, labelled by speaker by hand and stored as JSON. They were then scored for sentiment, emotion and valence with transformer models.
Audio: speech was segmented by speaker and analysed for valence and arousal.
Video: face-based emotion recognition was applied.

The main output is a practical "cookbook" workflow rather than a benchmark score. The authors also document the weaknesses of current models: age bias in facial recognition, weaker performance outside English, and no fair way to compare modalities. The work gives archives and researchers concrete steps for preparing heterogeneous testimony collections for multimodal emotion annotation.

Key points

Source data: 100 USHMM testimonies (30 Czech, 23 Polish, 20 English, 15 Dutch, 12 French), published as a multilingual dataset on Zenodo (Anuradha et al., 2026). The working subset had 10 interviews, 7 of them witnesses and 3 survivors.
Text models:
- Sentiment: cardiffnlp/twitter-roberta-base-sentiment (English) and nie3e/sentiment-polish-gpt2-large (Polish; its "ambiguous" outputs were dropped).
- Emotion: j-hartmann/emotion-english-roberta-large (English) and hplisiecki/polemo_intensity (Polish).
- Valence: chrlukas/stories-emotion-c0 (English). All valence scores were rescaled to the range −1 to 1.
Audio: speaker diarization and silence detection were performed. Valence and arousal came from a Wav2vec 2.0-based model following Wagner et al. (2023).
ASR: a Whisper model fine-tuned for French produced usable transcripts. Polish ASR performed poorly, so the study kept only existing English and Polish transcripts.
Video limitations: Emonet-style facial models, trained mostly on younger faces, often mislabelled elderly interviewees as sad or angry. Camera zooms, lighting changes and frames showing the interviewer added further noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓University of HelsinkiHelsinki ·

Finland50 works · 5,318 citations 🎓Technical University of MunichMunich ·

Germany55 works · 48,474 citations 🎓Universität TrierTrier ·

Germany8 works · 9 citations

Fields, topics and concepts

📚 Shelves 5Arts and Humanities · 16,039 works Psychology · 1,552 works Experimental and Cognitive Psychology · 870 works Social Psychology · 306 works Literature and Literary Theory · 519 works

🧭 Topics 2Sentiment & emotion · 972 works Vision & language · 657 works

Cite

Cite as Nele Mantaj, Vaibhav Agarwal and Ines Matres. 2026. Emotions In Oral History Interviews: A Multimodal Approach to Holocaust Testimonies. Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/59pkmnv7gtu2

@inproceedings{mantaj2026emotions,
  title = {{Emotions In Oral History Interviews: A Multimodal Approach to Holocaust Testimonies}},
  author = {Nele Mantaj, Vaibhav Agarwal and Ines Matres},
  booktitle = {Proceedings of The Second Workshop on Holocaust Testimonies as Language Resources (HTRes)},
  year = {2026},
  doi = {10.63317/59pkmnv7gtu2},
  url = {https://doi.org/10.63317/59pkmnv7gtu2}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W. Schuller. 2023. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):106… find
Isuri Anuradha, Le An Ha, Ruslan Mitkov, and Vinita Nahar. 2023. Evaluating of Large Language Models in Relationship Extraction from Unstructured Data: Empirical Study from Holocaust Testimonies. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, p… find
Zoë Waxman. 2012. Chapter 7. Transcending History? Methodological Problems in Holocaust Testimony. In Dan Stone, editor, The Holocaust and Historical Methodology, pages 143– 157. Berghahn Books. find
Isuri Anuradha Nanomi Arachchige, Le Ha, Ruslan Mitkov, and Johannes-Dieter Steinert. 2023. Enhancing Named Entity Recognition for Holocaust Testimonies through Pseudo Labelling and Transformer-based Models. In Proceedings of the 7th International Workshop on Historical Document Imaging and Processi… find
Data Sources Isuri Anuradha, Inés Matres, and Yu Wu. 2026. Multilingual dataset of interviews with survivors and witnesses of the holocaust https://doi.org/10.5281/zenodo.18701345. — [doi:10.5281/zenodo.18701345](https://doi.org/10.5281/zenodo.18701345) ↗
Hervé Bredin. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. find
Liudmila Bukreeva, Daria Guseva, Mikhail Dol-gushin, Vera Evdokimova, and Vasilisa Obot-nina. 2023. Emotional Speech Recognition of Holocaust Survivors with Deep Neural Network Models for Russian Language. In Alexey Karpov, K. Samudravijaya, K. T. Deepak, Rajesh M. Hegde, Shyam S. Agrawal, and S. R.… find
Nicole Eustace, Eugenia Lean, Julie Livingston, Jan Plamper, William M. Reddy, and Barbara H. find

Updated 1 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mantaj-2026-emotions-oral-history-interviews-multimodal/