📄 The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text

Summary

The paper presents the Fisher protocol, which the Linguistic Data Consortium (LDC) built for the DARPA EARS program. EARS needed far more transcribed conversational telephone speech (CTS) than earlier corpora offered, because CTS remained one of the hardest settings for ASR. Earlier corpora fell short in different ways. CALLFRIEND and Switchboard-II were designed for language and speaker identification. CALLHOME has few speakers, long calls and a narrow vocabulary.

Fisher reverses that design. It recruits a very large pool of speakers, each of whom makes only a few short calls with strangers on a topic assigned that day. Calls are mostly initiated by an automated platform, and recruitment and registration were largely automated as well.

In about eleven months the collection produced 16,454 English conversations, totaling 2742 hours. That exceeds the most ambitious target by 37%, and the speakers are better balanced by gender, age and dialect than in earlier corpora. A Quick Transcription specification allowed 2000 hours to be transcribed within a year. Fisher is already used in EARS training and in NIST's RT-03 evaluation, and LDC plans public release starting in 2004.

Key points

• Volume: 16,454 calls averaging about ten minutes, totaling 2742 hours. In its first cycle, Fisher yielded more data than LDC had gathered in the previous 12 years combined.
• Call design: The target was 2000 hours from calls of at most ten minutes. Roughly eight minutes per call was expected to be "useful," meaning on-topic speech without greetings and goodbyes.
• Demographics:
  • Women made 53% of calls, whereas past studies struggled to get below a 60/40 female/male split.
  • Ages: 38% were 16–29, 45% were 30–49 and 17% were over 50.
  • Recruitment followed Labov's four US dialect regions (North, Midland, South, West), and speakers from Canada, non-native speakers and other English varieties were also admitted.
• Collection rate: For nearly four months the system called only previously unseen speakers so that evaluation sets could be built. Lifting that limit raised throughput from about 15 to about 54 calls per day.
• Topics: An initial list of 40 topics was cycled roughly 8–9 times, with an average of about 400 calls per topic. A further 60 topics were each used once in late 2003. Calls per topic ranged from 207 to 648, mostly depending on how many weekend days a topic fell on.

Interesting ideas

• I1. Many speakers, few calls each: Limiting each participant to one to three short calls maximizes speaker variety and vocabulary spread. The paper notes the cost: speech becomes somewhat more formal. This trade-off applies to any speech or dialogue collection where coverage of speakers matters more than depth per speaker.
• I2. System-driven collection: The platform, not the participants, places most calls, at times the participants chose when registering. This removes dependence on volunteers taking the initiative and lets throughput scale with recruitment, so automated scheduling could speed up other crowd-based data gathering.
• I3. One shared daily topic, chosen at random: Every pair on a given day discusses the same randomly drawn topic, which keeps vocabulary broad while still grouping comparable conversations. Logging the weekday also helps explain why the amount of data per topic is uneven.
• I4. Reserving fresh speakers for test sets early: Calling only new subjects at the start produced evaluation sets with speakers who appear nowhere else in the corpus. This is a simple way to prevent speaker overlap between training and test data in new corpora.
• I5. Explicit definition of usable data: Setting targets in terms of on-topic minutes, with greetings and leave-takings excluded, makes collection goals measurable. This could be reused when planning budgets for annotated dialogue resources.