Elamur Research⌕
☰
home / papers / 2010 / Unsupervised Modeling of Twitter Conversations

Unsupervised Modeling of Twitter Conversations

Summary

The paper addresses how to find dialogue acts (shallow conversational roles such as questions or replies) in open-domain chat without a hand-built tag inventory or annotated data. The authors crawled Twitter for two months in summer 2009 and assembled about 1.3 million English reply-chain conversations, which they planned to release publicly. Their starting point is an HMM borrowed from Barzilay and Lee's content models: each post is emitted by one hidden act as a unigram bag of words, and acts follow a Markov chain. Trained with EM, this model tended to group conversations by topic (for example, food) rather than by dialogue function. A Conversation+Topic model fixes this with an LDA-style split: each word comes from the post's act, the conversation's topic, or general English. Bayesian variants are trained with Gibbs sampling. Evaluation uses visualization plus a new intrinsic task: recovering the original order of shuffled posts. Bayesian inference beat EM. The topic-aware model produced the clearest acts, but it was not the best at ordering. The work shows that dialogue structure can be learned quickly for a new medium.

Key points

  • Corpus: roughly 1.3M Twitter conversations, each with 2 to 243 posts. Two-post exchanges make up 69%, and conversation lengths follow a power law. Threads were rebuilt from reply links in the Twitter API, so no disentanglement step was needed.
  • Twitter text is very noisy, with even pronouns and prepositions spelled many ways. Jcluster word clustering exposed spelling variants such as b4/be4/befor.
  • Models compared: the EM Conversation model (an HMM trained with forward-backward), the Conversation+Topic model, and the Bayesian Conversation model. Each was tested with 5 to 40 acts.
  • Ordering evaluation:
    • Setup: 1,000 held-out conversations, with every n! permutation of each one scored and Kendall's τ computed against the true order.
    • Baseline: a bigram model reached only τ = 0.0358, compared with 0.19–0.74 reported on news data.
  • Further results:
    • Bayesian models achieved much higher held-out likelihood than EM, and EM overfit as the number of states grew.
    • The Bayesian Conversation model ordered posts better than Conversation+Topic, but the latter was far more interpretable.
    • Learned acts included Status, Question and Response, which differ in their use of the pronouns "I" and "you".

Interesting ideas

  • I1. Remove topic through the model, not the data. Giving each word a choice of source (act, topic, or background language) separates content from function. This could suit other noisy domains where masking words with NER or POS tools fails.
  • I2. Use reordering as a label-free evaluation. Asking a model to restore the order of shuffled units tests its sequential knowledge without any annotated tag set. The idea could transfer to evaluating discourse or document-structure models in low-resource languages.
  • I3. Report interpretability and prediction separately. The model that predicted order best was not the most interpretable one. Projects should therefore not rely on a single intrinsic metric when choosing models.
  • I4. Borrow models across genres. A summarization content model, with its hidden states reinterpreted as dialogue acts, became a dialogue model. Architectures from neighboring tasks are worth checking before designing new ones.
  • I5. Use platform metadata for structure. Reply links supplied clean conversation threads for free. Similar metadata on other platforms could avoid costly reconstruction steps when building corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

152 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 138 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ritter2010unsupervised,
  title = {{Unsupervised Modeling of Twitter Conversations}},
  author = {Ritter, Alan and Cherry, Colin and Dolan, Bill},
  booktitle = {Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/N10-1020/}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. 2007. Plow: a collaborative task learning agent. In Proceedings of AAAI. find
  2. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In Proceedings of HLT-NAACL, pages 113–120. find our card
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022. find our card
  4. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to classify email into “speech acts”. In Proceedings of EMNLP. find our card
  5. Nigel Crook, Ramon Granell, and Stephen Pulman. 2009. Unsupervised classification of dialogue acts using a Dirichlet process mixture model. In Proceedings of SIGDIAL, pages 341–348. find
  6. Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of ACL. find
  7. Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Elizabeth Shriberg. 2004. Meeting recorder project: Dialog act labeling guide. Technical report, International Computer Science Institute. find
  8. Micha Elsner and Eugene Charniak. 2008. You talking to me? A corpus and algorithm for conversation disentanglement. In Proceedings of ACL-HLT. find
  9. Eric N. Forsyth and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog. In Proceedings of ICSC. find
  10. Michael Gamon, Sumit Basu, Dmitriy Belenko, Danyel Fisher, Matthew Hurst, and Arnd Christian Knig. 2008. Blews: Using blogs to provide context for news articles. In Proceedings of ICWSM. find
  11. Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tagging. In Proceedings of ACL, pages 744–751. find our card
  12. Joshua T. Goodman. 2001. A bit of progress in language modeling. Technical report. find
  13. T. L. Griffiths and M. Steyvers. 2004. Finding scientific topics. Proc Natl Acad Sci, 101 Suppl 1:5228–5235. find
  14. Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of HLT-NAACL, pages 362–370. find our card
  15. Yijue How and Min-Yen Kan. 2005. Optimizing predictive text entry for short message service on mobile phones. In Proceedings of HCII. find
  16. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of EMNLP, pages 1250–1259. find
  17. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual, draft 13. Technical report, University of Colorado Institute of Cognitive Science. find
  18. Ryan Kelly. 2009. Pear analytics twitter study. Whitepaper, August. find
  19. Iain Murray and Ruslan Salakhutdinov. 2008. Evaluating probabilities under high-dimensional latent variable models. In Proceedings of NIPS, pages 1137– 1144. find
  20. Gabriel Murray, Steve Renals, Jean Carletta, and Johanna Moore. 2006. Incorporating speaker and discourse features into speech summarization. In Proceedings of HLT-NAACL, pages 367–374. find
  21. Radford M. Neal. 2003. Slice sampling. Annals of Statistics, 31:705–767. find
  22. Rajesh Ranganath, Dan Jurafsky, and Dan Mcfarland. 2009. It’s not you, it’s me: Detecting flirting and its misperception in speed-dates. In Proceedings of EMNLP, pages 334–342. find
  23. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. find our card
  24. Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David M. Mimno. 2009. Evaluation methods for topic models. In Proceedings of ICML, page 139. find
  25. Yorick Wilks. 2006. Artificial companions as a new kind of interface to the future internet. In OII Research Report No. 13. find
  26. M. Woszczyna and A. Waibel. 1994. Inferring linguistic structure in spoken language. In Proceedings of IC-SLP. find
  27. Tae Yano, William W. Cohen, and Noah A. Smith. 2009. Predicting response to political blog posts with topic models. In Proceedings of NAACL, pages 477–485. find
  28. Limin Yao, David Mimno, and Andrew McCallum. 2009. Efficient methods for topic model inference on streaming document collections. In Proceedings of KDD, pages 937–946. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/