📄 Not All Polar Questions Are the Same: ASR, Humans, and Russian

Summary

WER measures whether an ASR system gets the words right, but it says nothing about prosody. This study checks whether current models detect a distinction that Russian can signal through intonation alone: polar questions versus assertions. Russian yes/no questions can carry the particles li or razve, or keep declarative word order and rely only on an L+H* accent on the verb (intonPQs). The study reused audio stimuli from an eye-tracking experiment and treated the final punctuation mark in each transcript as a sign of how the model interpreted the utterance's prosody. The audio was transcribed by Whisper, Meta's MMS and two GigaAM variants, and the results were compared with a human forced-choice task. Humans were almost perfect. The models handled assertions and razvePQs well but often missed intonPQs, and some showed strong asymmetries between positive and negative items. Adding question-related context usually helped, while unrelated context gave unstable results. The findings argue for punctuation-based evaluation as a complement to WER when probing prosodic and grammatical knowledge in ASR.

Key points

• Models and setup: Whisper large_v3 (greedy decoding), MMS (CTC, with punctuation restored by the Silero Text Enhancement model) and GigaAM in CTC and RNNT variants. No other external language models were used.

• Humans versus models on plain audio: Human listeners (N=54) classified the plain recordings with 97–100% accuracy. For intonPQs, model accuracy was much lower:

  Model	Negative intonPQs	Positive intonPQs
  MMS	0%	0%
  Whisper	62.5%	6.2%
  GigaAM RNNT	100%	34.4%
  GigaAM CTC	96.9%	56.2%

• Particle-marked questions: Questions marked with li or razve were recognised more reliably than purely intonational ones. MMS was the weak exception, reaching only 18.8% on liPQs and 68.8% on razvePQs.

• Enriched audio: Nine extra contexts, recorded by a female speaker, were added before or after the original utterances. Question-related cues (e.g. "The next sentence is a question", "He asked", or a following yes/no reply) significantly improved intonPQ detection. With a "yes" reply, Whisper reached 90.6% on positive intonPQs, and GigaAM CTC reached 87.5–100%.

• Unrelated context: Neutral additions such as "turtle", "He said" or Pushkin excerpts sometimes harmed performance. With a preceding "turtle", GigaAM RNNT fell to 3.1% on positive intonPQs.