Elamur Research
☰
home / papers / 2012 / Sequence Transduction with Recurrent Neural Networks

Sequence Transduction with Recurrent Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1211.3711

Summary

Ordinary recurrent networks can map one sequence to another only when the positions of inputs and outputs are matched in advance. In tasks such as speech recognition, finding that match, and even the output length, is often the hardest part. Graves proposes the RNN Transducer, a fully neural, probabilistic model that can map any input sequence to any finite discrete output sequence. It combines two parts: a bidirectional LSTM transcription network that reads the input, and an LSTM prediction network that works like a language model over earlier outputs. Their outputs are merged at every pair of input and output positions. A special null symbol lets the model emit nothing at a given step, so all alignments form a lattice that a forward-backward procedure sums efficiently. Decoding uses beam search. Unlike CTC, the model captures dependencies between output labels and permits outputs longer than inputs. On TIMIT phoneme recognition it reaches 23.2% error, beating a CTC baseline and, the author reports, the best RNN result at that time. The design later became a basis for streaming speech recognition.

Key points

  • Data: The TIMIT core sets were used, with 3696 training and 192 test utterances. A 184-utterance validation set was taken out of the training data. The targets were the reduced inventory of 39 phonemes.
  • Inputs: The input features were 26-dimensional, consisting of 12 MFCCs plus energy together with their deltas, computed on 25 ms windows every 10 ms.
  • Architecture: The transcription network has two 128-unit LSTM layers, and the prediction network has one 128-unit LSTM layer. Each network has 40 outputs, which are the 39 phonemes plus null. The full transducer has 261,328 weights.
  • Results (test phoneme error / log-loss in bits per phoneme):
    • Transducer: 23.2% / 1.0
    • CTC: 25.5% / 1.3
    • Standalone prediction network: 72.9% / 4.0
    • The state-of-the-art benchmark at the time was 20.5% (Dahl et al., 2010).
  • Why the gain is small: The author attributes the modest improvement over CTC to the small amount of label data, about 150K phonemes. He suggests pretraining the prediction network on large target-only corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

225 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 216 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1211.3711 · arXiv 1211.3711

@misc{graves2012sequence,
  title = {{Sequence Transduction with Recurrent Neural Networks}},
  author = {Graves, Alex},
  year = {2012},
  doi = {10.48550/arxiv.1211.3711},
  eprint = {1211.3711},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1211.3711}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bertolami, R, Zimmermann, M, and Bunke, H. Rejection strategies for offline handwritten text line recognition. Pattern Recognition Letters, 27(16): 2005–2012, 2006. find
  2. Bottou, Leon, Bengio, Yoshua, and Cun, Yann Le. Global training of document processing systems using graph transformer networks. In CVPR, pp. 489–, 1997. find
  3. Dahl, G., Ranzato, M., Mohamed, A., and Hinton, G. Phone recognition with the mean-covariance restricted boltzmann machine. In NIPS. 2010. find
  4. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus (TIMIT). DARPA-ISTO, speech disc cd1-1.1 edition, 1990. find
  5. Gers, F. Long Short-Term Memory in Recurrent Neural Networks. PhD thesis, Ecole Polytechnique Fédérale de Lausanne, 2001. find
  6. Graves, A. and Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18: 602–610, 2005. find
  7. Graves, A. and Schmidhuber, J. Offline handwriting recognition with multidimensional recurrent neural networks. In NIPS, 2008. find
  8. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006. find
  9. Graves, A., Fernández, S., Liwicki, M., Bunke, H., and Schmidhuber, J. Unconstrained Online Handwriting Recognition with Recurrent Neural Networks. In NIPS. 2008. find
  10. Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 9(8):1735– 1780, 1997. find
  11. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-term Dependencies. In Kremer, S. C. and Kolen, J. F. (eds.), A Field Guide to Dynamical Recurrent Neural Networks. 2001. find
  12. Jim, Kam-Chuen, Giles, C., and Horne, B. An analysis of noise in recurrent neural networks: convergence and generalization. Neural Networks, IEEE Transactions on, 7(6):1424 –1438, 1996. find
  13. Lafferty, J. D., McCallum, A., and Pereira, F. C. N. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In ICML, 2001. find
  14. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  15. Lee, K. and Hon, H. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1989. find
  16. Mikolov, T., Karafit, M., Burget, L., Cernocky, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  17. Schuster, M. and Paliwal, K. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673–2681, 1997. find
  18. Sutskever, I., Martens, J., and Hinton, G. Generating text with recurrent neural networks. In ICML, 2011. find
  19. Williams, R. and Zipser, D. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Back-propagation: Theory, Architectures and Applications, pp. 433–486. 1995. find