Elamur Research⌕
☰
home / papers / 2014 / Dynamic Language Models for Streaming Text

Dynamic Language Models for Streaming Text

Paper · 2014 · Dani Yogatama, Chong Wang, Bryan R. Routledge et al.

Summary

Most language models assume word distributions stay fixed over time. That assumption fits poorly with fast-changing streams like news and social media. Yogatama et al. propose a probabilistic language model whose parameters evolve daily and can depend on observable non-text signals, here stock market movements. Each day's word distribution combines background log-frequencies with a deviation vector. That vector is a weighted mixture of deviations from a fixed window of recent days. The weights reflect how closely current market conditions resemble those earlier days, scored by learned feature coefficients. Training is fully online. It combines mean-field variational inference with ideas from regret-minimizing sequential prediction, so old data need not be stored. The authors define a new task: predicting a day's economics-related text given stock prices up to that day. They test unigram and bigram base models on five streams: Reuters economic news in English, French and German, plus Twitter. Their model beats competing baselines in perplexity across these streams. The work offers a scalable way to link text modeling with numeric real-world context.

Key points

  • Data: four Reuters news streams (EN:NA, EN:JP, FR, DE) and one Twitter stream, with stock price changes as context features.
  • The default history window is c = 7 days. Expanding it to c = 14 brought no gain, and the model selects the t − 1 day's distribution most often.
  • On Twitter, deviation coefficients for terms like goog, #goog, @google and #microsoft rise around sharp moves in Google and Microsoft stock returns. Highly weighted stocks include ConAgra Foods, Intel and Amazon.com.
  • The approach works with any base model. Bigrams are handled with one β per history word, and the method relates to dynamic topic models and the online model of Iwata et al. (2010).
  • Single-membership topic variables did not improve perplexity and raised computational cost. LDA-style extensions are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith and Eric P. Xing. 2014. Dynamic Language Models for Streaming Text. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00175

@article{yogatama2014dynamic,
  title = {{Dynamic Language Models for Streaming Text}},
  author = {Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith and Eric P. Xing},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00175},
  url = {https://doi.org/10.1162/tacl_a_00175}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Galen Andrew and Jianfeng Gao. 2007. Scalable training of l 1 -regularized log-linear models. In Proc. of ICML. find
  2. David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In Proc. of ICML.
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. Sébastien Bubeck. 2011. Introduction to online optimization. Technical report, Department of Operations Research and Financial Engineering, Princeton University.…
  4. Prediction, Learning, and Games. Cambridge University Press. William S. Cleveland. 1979. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368):829–836. find
  5. Thomas M. Cover and Joy A. Thomas. 1991. Elements of Information Theory. find
  6. John Wiley & Sons. John Duchi and Yoram Singer. 2009. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10(7):2899– 2934. find
  7. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Proc. of EMNLP.
  8. Jacob Eisenstein, Amr Ahmed, and Eric P. Xing. 2011. Sparse additive generative models of text. In Proc. of ICML. find
  9. Amit Goyal, Hal Daume III, and Suresh Venkatasubramanian. 2009. Streaming for large scale NLP: Language modeling. In Proc. of HLT-NAACL. find
  10. Matt Hoffman, David M. Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. Journal of Machine Learning Research, 14:1303–1347. find
  11. Antti Honkela and Harri Valpola. 2003. On-line variational Bayesian learning. In Proc. of ICA. find
  12. Tomoharu Iwata, Takeshi Yamada, Yasushi Sakurai, and Naonori Ueda. 2010. Online multiscale dynamic topic models. In Proc. of KDD. find
  13. Frederick Jelinek. 1997. Statistical Methods for Speech Recognition. find
  14. MIT Press. Jyrki Kivinen and Manfred K. Warmuth. 1997. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132:1–63. find
  15. Victor Lavrenko, Matt Schmill, Dawn Lawrie, Paul Ogilvie, David Jensen, and James Allan. 2000. Mining of concurrent text and time series. In Proc. of KDD Workshop on Text Mining. find
  16. Abby Levenberg and Miles Osborne. 2009. Stream-based randomised language models for SMT. In Proc. of EMNLP. find
  17. Abby Levenberg, Chris Callison-Burch, and Miles Osborne. 2010. Stream-based translation models for statistical machine translation. In Proc. of HLT-NAACL. find
  18. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical Programming B, 45(3):503–528. find
  19. David Mimno and Andrew McCallum. 2008. Topic models conditioned on arbitrary features with Dirichlet-multinomial regression. In Proc. of UAI. find
  20. Alexander Rakhlin. 2009. Lecture notes on online learning. Technical report, Department of Statistics, The Wharton School, University of Pennsylvania. find
  21. Masaaki Sato. 2001. Online model selection based on the variational bayes. Neural Computation, 13(7):1649– 1681. find
  22. Shai Shalev-Shwartz. 2012. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107–194. find
  23. Martin J. Wainwright and Michael I. Jordan. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305. find
  24. Chong Wang, David M. Blei, and David Heckerman. 2008. Continuous time dynamic topic models. In Proc. of UAI. find
  25. Chong Wang, John Paisley, and David M. Blei. 2011. Online variational inference for the hierarchical Dirichlet process. In Proc. of AISTATS. find
  26. Martin Zinkevich. 2003. Online convex programming and generalized infinitesimal gradient ascent. In Proc. of ICML. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yogatama-2014-dynamic-language-models-streaming-text/