Elamur Research⌕
☰
home / papers / 2018 / Algorithmes à base d'échantillonage pour l'entraînement de modèles de langue neuronaux (Here the title in English)

Algorithmes à base d'échantillonage pour l'entraînement de modèles de langue neuronaux (Here the title in English)

Paper · 2018 · Matthieu Labeau and Alexandre Allauzen

Summary

Neural language models must turn output scores into probabilities over the whole vocabulary. With large vocabularies, computing this softmax normalization, and its gradient, dominates training cost. This paper from LIMSI first surveys the main sampling-based ways to avoid that cost. It covers Importance Sampling (IS) and its target-sampling variant, Noise Contrastive Estimation (NCE), BlackOut, and Negative Sampling (NS), and explains how each replaces the expensive normalization term with estimates computed from a few sampled words. It also reviews how these methods have been used in machine translation and speech recognition. The authors then compare the algorithms experimentally on two reference language-modelling corpora. One setup uses a small corpus and a closely monitored model. The other uses a larger corpus, where full maximum likelihood would be too expensive. Their working hypothesis is that NCE is hard to train because the model must learn to self-normalize as a side task. They therefore test heuristics that ease this, measured against an NCE model whose scores are explicitly normalized. The detailed numbers are in the part of the text not available here. The paper is useful as a compact guide to choosing a sampling objective for large-output models.

Key points

  • IS approximates the "push down all other words" part of the gradient with weighted samples from an easy proposal distribution. This gives a biased estimator whose weights can become very large and inflate variance. Target sampling (Jean et al., 2015) avoids this by sampling uniformly from data-partition subsets that always contain the gold words.
  • NCE turns density estimation into a binary task of telling data from noise. Its weights stay between 0 and 1. With enough noise samples, the estimation error becomes asymptotically independent of the noise distribution. Mnih & Teh (2012) found unigram noise works far better than uniform noise.
  • NS, as in skip-gram, does not directly optimize language-model likelihood. Its score models a log-ratio against the noise distribution, and a probability can be recovered as P_n(w)·exp(s).
  • Prior NCE work in MT and ASR fixed the log-partition constant at 9 rather than 1 to make training easier.
  • Chen et al. (2016) reported that NCE can be inconsistent on very large vocabularies, and Józefowicz et al. (2016) found it less effective than IS.

Interesting ideas

  • I1. The authors train an NCE model whose scores are explicitly normalized. It is useless in practice, but it shows how much of the difficulty comes from self-normalization alone and so gives a clean reference for judging fixes.
  • I2. Target sampling computes the softmax only over a sampled candidate set that is guaranteed to contain the correct answer. This keeps the estimator consistent while cutting cost.
  • I3. Choosing the noise distribution is a trade-off between resembling the real data and being cheap enough to draw many samples. More samples can compensate for a cruder noise source.
  • I4. Bounded weights, as in NCE, avoid the variance blow-up that unbounded importance ratios cause in IS.
  • I5. Negative sampling learns a score that is a log-ratio of data probability to noise probability, a PMI-like quantity, from which a probability can be recovered by multiplying by the noise distribution.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When evaluating a cheap training shortcut, also train a "costly but clean" variant that removes one suspected difficulty, so the shortcut's loss can be attributed to a specific cause.
  • I2 (tree): gparse could score dependency labels or head candidates over a pruned set that always includes the gold option during training, which saves GPU time without biasing toward impossible choices.
  • I3 (learn): When building negative controls or contrastive negatives for training, prefer many cheap, frequency-shaped negatives over a few expensive hand-crafted ones, and track how results shift as the count grows.
  • I4 (learn): If silver examples are weighted by how strongly witness parsers agree, cap the weights so a few extreme examples cannot dominate the gradient.
  • I5 (frames): Verb–argument plausibility for VerbNet role assignment could be learned as a contrastive log-ratio against frequency-based noise, giving an interpretable association score rather than a raw probability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Matthieu Labeau and Alexandre Allauzen. 2018. Algorithmes à base d'échantillonage pour l'entraînement de modèles de langue neuronaux (Here the title in English). Actes de la Conférence TALN. Volume 1 - Articles longs, articles courts de TALN.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{labeau2018algorithmes,
  title = {{Algorithmes à base d'échantillonage pour l'entraînement de modèles de langue neuronaux (Here the title in English)}},
  author = {Matthieu Labeau and Alexandre Allauzen},
  booktitle = {Actes de la Conférence TALN. Volume 1 - Articles longs, articles courts de TALN},
  year = {2018},
  url = {https://aclanthology.org/2018.jeptalnrecital-court.29/}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A NDREAS J., R ABINOVICH M., J ORDAN M. I. & K LEIN D. (2015). On the accuracy of self-normalized log-linear models. In Advances in Neural Information Processing Systems 28 : Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, p. 1783–1791. find
  2. B ALTESCU P. & B LUNSOM P. (2015). Pragmatic neural language modelling in machine translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, p. 820–829, Denver, Colorado : Association for Computational Linguistics. find
  3. B ENGIO Y., D UCHARME R. & V INCENT P. (2001). A neural probabilistic language model.
  4. B ENGIO Y. & S ÉNÉCAL J.-S. (2003). Quick training of probabilistic neural nets by importance sampling. In Proceedings of the conference on Artificial Intelligence and Statistics (AISTATS). find
  5. B ENGIO Y. & S ÉNÉCAL J.-S. (2008). Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Trans. Neural Networks, 19(4), 713–722. find
  6. C HELBA C., M IKOLOV T., S CHUSTER M., G E Q., B RANTS T., K OEHN P. & R OBINSON T. (2014). One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014, p. 2635–2639.
  7. C HEN W., G RANGIER D. & A ULI M. (2016). Strategies for training large vocabulary neural language models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers), p. 1975–1985, Berlin, Germany : Association for Computational Linguistics. find
  8. C HEN X., L IU X., G ALES M. J. F. & W OODLAND P. C. (2015). Recurrent neural network language model training with noise contrastive estimation for speech recognition. In ICASSP, p. 5411–5415 : IEEE. find
  9. D EVLIN J., Z BIB R., H UANG Z., L AMAR T., S CHWARTZ R. & M AKHOUL J. (2014). Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers), p. 1370–1380, Baltimore, Maryland : Association for Computational Linguistics.
  10. D YER C. (2014). abs/1410.8251. Notes on noise contrastive estimation and negative sampling. ↗
  11. CoRR, G UTMANN M. & H YVÄRINEN A. (2010). Noise-contrastive estimation : A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, p. 297–304. find
  12. G UTMANN M. & H YVÄRINEN A. (2012). Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13, 307–361. find
  13. G UTMANN M. & H YVÄRINEN A. (2013). Estimation of unnormalized statistical models without numerical integration. In Proceedings of the Workshop on Information Theoretic Methods in Science and Engineering. find
  14. J EAN S., C HO K., M EMISEVIC R. & B ENGIO Y. (2015). On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1 : Long Papers), p. 1–10, Beijing, China : Association for Computational Linguistics.
  15. J I S., V ISHWANATHAN S. V. N., S ATISH N., A NDERSON M. J. & D UBEY P. (2015). Blackout : Speeding up recurrent neural network language models with very large vocabularies. CoRR, abs/1511.06909. ↗
  16. J ÓZEFOWICZ R., V INYALS O., S CHUSTER M., S HAZEER N. & W U Y. (2016). Exploring the limits of language modeling. CoRR, abs/1602.02410. ↗
  17. K INGMA D. P. & B A J. (2014). abs/1412.6980. Adam : A method for stochastic optimization. ↗
  18. CoRR, L E H.-S., O PARIN I., A LLAUZEN A., G AUVAIN J.-L. & Y VON F. (2011). Structured output layer neural network language model. p. 5524–5527, Prague, Czech Republic. find
  19. M ELAMUD O., D AGAN I. & G OLDBERGER J. (2016). PMI matrix approximations with applications to neural language modeling. CoRR, abs/1609.01235. ↗
  20. M ELAMUD O., D AGAN I. & G OLDBERGER J. (2017). A simple language model based on pmi matrix approximations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 1861–1866, Copenhagen, Denmark : Association for Computational Linguistics. find
  21. M IKOLOV T., K ARAFIÁT M., B URGET L., C ERNOCKÝ J. & K HUDANPUR S. (2010). Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, p. 1045–1048. find
  22. M IKOLOV T., S UTSKEVER I., C HEN K., C ORRADO G. S. & D EAN J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. B URGES , L. B OTTOU , M. W ELLING , Z. G HAHRAMANI & K. Q. W EINBERGER , Eds., Advances in Neural Information Processing Systems 26, p. 3111–3119. Curran Associates, Inc.
  23. M NIH A. & H INTON G. E. (2009). A scalable hierarchical distributed language model. In D. K OLLER , D. S CHUURMANS , Y. B ENGIO & L. B OTTOU , Eds., Advances in Neural Information Processing Systems 21, p. 1081–1088. Curran Associates, Inc. find
  24. M NIH A. & T EH Y. W. (2012). A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012.
  25. M ORIN F. & B ENGIO Y. (2005). Hierarchical probabilistic neural network language model. In R. G. C OWELL & Z. G HAHRAMANI , Eds., Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, p. 246–252 : Society for Artificial Intelligence and Statistics. find
  26. S CHWENK H. (2007). Continuous space language models. Comput. Speech Lang., 21(3), 492–518. find
  27. V ASWANI A., Z HAO Y., F OSSUM V. & C HIANG D. (2013). Decoding with large-scale neural language models improves translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, p. 1387–1392, Seattle, Washington, USA : Association for Computational Linguistics.
  28. Z OPH B., V ASWANI A., M AY J. & K NIGHT K. (2016). Simple, fast noise-contrastive estimation for large rnn vocabularies. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, p. 1217–1222, San Diego, California : Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/labeau-2018-algorithmes-base-d-echantillonage-pour/