📄 Call Support Copilot: A Reproducible Multimodal System for Speech Emotion Recognition, Intent Understanding, and Agent Assistance

Summary

Customer support agents must follow what a caller says, gauge their mood, and look up account data all at once. Prior work mostly evaluates the relevant technologies in isolation, and such systems are often hard to reproduce. The authors build Call Support Copilot, a Streamlit dashboard that chains public pre-trained checkpoints. faster-whisper ASR produces transcripts, which go to MarianMT for translation into English and to a DistilBERT intent classifier fine-tuned on BANKING77, with regex-based slot filling. In parallel, MERaLiON-SER-v1 tracks caller emotion as categorical labels plus valence-arousal-dominance scores over sliding windows. These signals are combined with records from an SQLite client database to suggest agent actions, such as follow-up questions or de-escalation tips. No ASR, MT or SER model is trained from scratch. Each module is benchmarked separately on standard data, and pipeline speed is measured. The intent classifier slightly outperforms a published BANKING77 baseline, and processing runs faster than real time. The authors note caveats: the emotion score rests on only six utterances, ASR error propagation is not measured, and there is no user study. The main contribution is a fully documented, auditable recipe.

Key points

• Component results:
  • 6.6% WER on 50 LibriSpeech test-clean samples (Systran/faster-whisper-base, int8).
  • 42.98 BLEU on 50 WMT16 German–English pairs (Helsinki-NLP opus-mt models).
  • 87.0% accuracy and 0.863 macro-F1 on BANKING77, versus 85.19% for USE+ConveRT (Casanueva et al., 2020).
• The SER figure of 91.7% macro-F1 comes from the SUPERB ER session1 split (IEMOCAP-derived, four classes, n=6). The authors treat it only as a sanity check.
• Mean real-time factor is 0.71 on LibriSpeech (n=10) and 0.67 on SUPERB ER (n=6). Sliding-window SER (4 s windows, 2 s overlap) accounts for most of the latency.
• Translation covers six European source languages into English: German, Dutch, French, Spanish, Italian and Portuguese. The system handles uploaded recordings only, not live streams.
• Model IDs, quantization settings, dataset slices, scripts and CSV/JSON outputs are released at github.com/rakhmatillokhon-khoshimov/call-support-copilot.