📄 How Does a Deep Neural Network Look at Lexical Stress in English Words?

Summary

Neural networks detect lexical stress well, but it is unclear which acoustic evidence drives their decisions. The authors address this for English disyllabic words. They built a dataset with no human annotation of stress. ChatGPT 4o supplied word lists: 25 noun/verb stress minimal pairs such as protest, and 249 words without a minimal-pair counterpart. Tokens were cut from LibriSpeech, the Supreme Court corpus and TED-LIUM using forced alignment. Stress labels came from SpaCy part-of-speech tags for minimal pairs (noun means initial stress, verb means final stress) and from the CMU Pronouncing Dictionary for the other words. Several CNNs were trained on spectrograms of the non-minimal-pair words only and reached up to 92% accuracy on held-out data. Layerwise Relevance Propagation (LRP), applied to the unseen minimal pairs, shows the networks rely mostly on the stressed syllable, especially its vowel, while still using cues spread across the whole word. A new feature-specific relevance analysis points to F1 and F2 of the stressed vowel as the dominant cues, with some role for pitch and F3. The work shows that deep models learn distributed stress cues from natural, uncontrolled speech.

Key points

• The dataset is released as MLSpeech/lexical_stress_dataset on Hugging Face. It has 7,446 initial-stress samples (124 types) and 3,263 final-stress samples (125 types) among the no-minimal-pair words.
• Alignment used the Montreal Forced Aligner for LibriSpeech and Supreme Court, and the provided timestamps for TED-LIUM. Each word sits in a fixed 0.5 s zero-padded window, and syllables were segmented with the P2TK syllabifier.
• The POS tagger mislabelled address, content, export, extract and decrease often enough that every instance of these words was manually relabelled.
• Training and test vocabularies were kept apart: models learned from non-minimal-pair words and were evaluated on minimal pairs they had never seen. This addresses lexical overlap left undocumented in earlier work.
• The contribution of syllabic and sub-syllabic regions to LRP heatmaps was quantified with an IOU metric. Feature-specific heatmaps for the stressed vowel were then compared against the full heatmap.

Interesting ideas

• I1. Holding out whole word types, here the hardest ambiguous ones, keeps memorised vocabulary from inflating accuracy and makes the test a real check of generalisation.
• I2. Labels were derived automatically from another system's output (POS → stress), and human review was spent only on the few word types where that proxy failed systematically.
• I3. Attribution maps were scored against linguistically defined spans (syllables, vowels) with IOU, turning a vague heatmap into a measurable claim about where the model looks.
• I4. Building a relevance map restricted to one hypothesised cue and comparing it with the overall relevance tests whether the model uses known, theory-backed features.
• I5. Noun/verb stress homographs such as protest and record exposed weaknesses in an off-the-shelf POS tagger.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Build evaluation splits for the tagger and parsers where test lemmas or constructions never occur in training, as a negative control against memorisation.
• I2 (learn): When creating silver data from witness parsers, group disagreements by lemma or type and send only the clusters with systematic errors to the LLM teacher or manual review.
• I3 (tree): Score gparse attributions against expected spans, such as the candidate head and its dependent, to check that attachment decisions rest on the right words.
• I4 (tree): Restrict attribution to a single input feature group (morphology, POS, word form) to test whether gparse relies on interpretable cues rather than lexical identity.
• I5 (tokens): Keep a targeted regression set of noun/verb homographs for the tagger, since errors there spread into the tree and frames.