📄 A Corpus of Spontaneous L2 English Speech for Real-situation Speaking Assessment

Summary

Automated speaking-assessment tools are usually built on tightly controlled tasks, such as reading words or sentences aloud, so they say little about how learners perform in real conversation. The authors address this with a 20-hour corpus of spontaneous L2 English recorded during CLES certification exams at Université Grenoble Alpes. In the exam, 260 students argue opposing sides of a contested topic and negotiate a compromise, and experts rate each one as B2, B1 or non-validated. Most of the corpus is publicly released. As a first use of the data, the authors built an open, interpretable pipeline: diarization, then ASR with word alignment, then syllable-nucleus detection, then speaker-normalised prosody. They used it to check whether 176 French-L1 speakers put primary stress where the CMU dictionary expects it. Only about a third of the words carried stress on the right syllable, and learners tended to shift prominence to the last syllable, a pattern consistent with French. B2 speakers did slightly better than B1 speakers. The work matters because it supplies real-situation data and transparent rhythm measures that current scoring systems rarely include.

Key points

• Corpus scope: 260 speakers in 134 recording groups, 20 hours in total.
  • Mostly 2-speaker B2-level role-plays of about 10 minutes each.
  • A few 3-speaker conversations and 13 B1-level monologues.
  • 83% of speakers have French as L1.
  • The public portion (128 students) is available at hdl.handle.net/11403/cles-spontaneous-english.
• Level split: 58% B2, 29% B1, 13% non-validated.
  • B2 raters score eight dimensions.
  • Failing any one dimension drops the candidate to B1 or no validation.
• Pipeline:
  • Pyannote for diarization.
  • WhisperX for transcription and word alignment.
  • SpaCy for POS tagging.
  • A Praat syllable-nuclei script with a 300–3300 Hz bandpass filter.
  • CMU Pronouncing Dictionary for expected stress.
  • The pipeline is released as open source (plspp).
• Main result: 35.4% of 6,350 target words were stressed correctly.
  • Per-speaker accuracy ranged from 0% to 68.4%.
  • Median accuracy was 36% for B2 and 29.6% for B1 (p < .0001).
  • Accuracy above 50% was reached by 22% of B2 speakers but only 3% of B1 speakers.
• How prominence is made: accurate speakers marked stress with F0 and intensity, while weak ones lengthened final syllables.
  • Stressed–unstressed contrast correlates with accuracy (R = .82).
  • A manual check of 100 random words found 92 correctly recognised and 95 correctly aligned.

Interesting ideas

• I1. Self-consistency filter: a word was kept only if the dictionary's syllable count matched the number of nuclei detected inside its aligned boundaries, so misaligned items are dropped without any gold annotation. This is a cheap internal check that guards automatic pipelines against garbage input.
• I2. Per-speaker percentiles: every prosodic value was converted to a percentile within that speaker's own distribution, which removes differences in speech rate, voice and microphone. Normalising each source against itself makes heterogeneous inputs comparable.
• I3. Expected vs. observed diagnosis: the authors deliberately avoided a trained black-box classifier. They compared what each speaker produced against a reference lexicon pattern and read off how errors arise (duration vs. F0/intensity), which explains a systematic L1-driven shift rather than just scoring it.
• I4. Small random audit: pipeline quality was estimated by hand-checking 100 randomly sampled words, separately for recognition and for alignment. Such a cheap per-stage audit can show which step is the weak link.
• I5. Pooling cues with equal weight: stress was decided from F0, intensity and duration combined with equal weight, and each cue was also inspected separately. This keeps the aggregate decision simple while still allowing per-cue explanation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when building silver data, keep only sentences where independent signals agree (for example, token counts or a tagger's POS against the parser's head structure), and reject the rest instead of training on them.
• I2 (learn): express parser confidence or feature scores as percentiles within each witness parser or each document genre, so that thresholds for agreement do not depend on one parser's scale.
• I3 (answer): grade student or learner answers by aligning the produced frame against the expected frame and reporting which role or relation diverged, giving an explainable diagnosis instead of a single score.
• I4 (learn): add per-stage random audits (for example, 100 tokens for tagging, 100 arcs for attachment, 100 frames for roles) to locate where errors enter the text → answer chain.
• I5 (frames): combine evidence for a speech-act or irony decision from several interpretable cues with transparent weights, and expose each cue's contribution in the explanation.