Elamur Research⌕
☰
home / papers / 2011 / Data-Driven Response Generation in Social Media

Data-Driven Response Generation in Social Media

Summary

The paper asks whether an open-domain system can reply to a Twitter status post by "translating" it into a response. The authors treat roughly 1.3 million Twitter conversations, cut down to status–reply pairs, as a parallel corpus and train a phrase-based SMT system with the Moses decoder. They find that naive MT mostly echoes the input. Standard word alignment also breaks down, because replies share few aligned words with the stimulus and often correspond to it only through large, indivisible chunks.

To fix the echoing, they remove phrase pairs where one side is a substring of the other and add a Jaccard-based penalty for lexical overlap. To replace noisy GIZA++ alignments, they enumerate all candidate phrase pairs and keep only those that pass Fisher's Exact Test. Mechanical Turk judges preferred the resulting system, MT-CHAT, over retrieval-based baselines, and chose it over real human replies 15% of the time. The work is an early demonstration that data-driven generation, with no templates, can produce plausible conversational responses.

Key points

  • Data: about 1.3M Twitter conversations, limited to the first two turns (status plus first reply). The resulting system is therefore specialized for answering status posts.
  • Alignment is very sparse. Intersected bidirectional GIZA++ gives only about 1.7 links per pair, compared with roughly 14 on WMT 08 French–English news commentary.
  • In human evaluation, MT-CHAT beat the IR approaches, which pick an existing response from the training data. It was preferred over real human replies in 15% of 200 filtered on-topic cases.
  • Direct phrase extraction with Fisher's Exact Test beat the standard Moses extraction pipeline (MT-BASELINE), winning 58% of comparisons.
  • Smoothed BLEU scores were very low. Even so, they matched the overall human rankings in every comparison except MT-CHAT vs. IR-RESPONSE, which suggests only limited usefulness as a metric.

Interesting ideas

  • I1. Anti-copying constraints for same-language "translation": drop phrase pairs where one side contains the other, and penalize word-set (Jaccard) overlap. This could help any monolingual seq2seq task where models tend to parrot the input, such as paraphrasing, simplification, or reply suggestion.
  • I2. Skip word alignment and filter phrase pairs statistically instead: treat every phrase pair as a candidate and keep those whose co-occurrence is significant under Fisher's Exact Test. This is useful wherever alignments are loose or non-compositional, for example noisy comparable corpora or low-resource Ukrainian data.
  • I3. Mining platform reply structure as free parallel data: reply links from an API give ready-made stimulus–response pairs with no need to untangle threads. The same trick could be applied to forums or comment sections to build conversational corpora.
  • I4. Evaluating against real humans and a "generic reply" baseline: comparing against real human answers (with off-topic ones filtered out) and against frequent stock replies tests whether outputs are actually tailored to the input. This is a cheap sanity check for any generation system.
  • I5. Checking whether an automatic metric fits a new task: before relying on BLEU, the authors measured how well it agreed with pairwise human preferences. Doing the same is good practice before adopting any metric in a new domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

164 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ritter2011data,
  title = {{Data-Driven Response Generation in Social Media}},
  author = {Ritter, Alan and Cherry, Colin and Dolan, William B.},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1054/}
}

References

46 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Comput. Linguist., 34:555–596, December. find
  2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: an entity-based approach. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05. find
  3. E. M. Bennett, R. Alpert, and A. C. Goldstein. 1954. Communications through limited-response questioning. Public Opinion Quarterly, 18(3):303–308. find
  4. Michael Bloodgood and Chris Callison-Burch. 2010. Using mechanical turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pages 208–211, Morristown, NJ, USA. Association for Com… find
  5. Chris Brockett. 2006. Aligning the rte 2006 corpus. In Microsoft Research Techincal report MSR-TR-2007-77. find
  6. Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. 1990. A statistical approach to machine translation. Comput. Linguist., 16:79–85, June. find
  7. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT ’09. find our card
  8. Nathanael Chambers and James Allen. 2004. Stochastic language generation in a dialogue system: Toward a domain independent generator. In Michael Strube and Candy Sidner, editors, Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 9– 18, Cambridge, Massachusetts, USA, April 30 -… find
  9. Cristian Danescu-Niculescu-Mizil, Michael Gamon, and Susan Dumais. 2011. Mark my words! Linguistic style accommodation in social media. In Proceedings of WWW. find
  10. Hal Daumé III and Daniel Marcu. 2009. Induction of word and phrase alignments for automatic document summarization. CoRR, abs/0907.0804. — [arXiv:0907.0804](https://arxiv.org/abs/0907.0804) find
  11. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, COLING ’04, Morristown, NJ, USA. Association for Computational Lingu… find our card
  12. Abdessamad Echihabi and Daniel Marcu. 2003. A noisy-channel approach to question answering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 16–23, Morristown, NJ, USA. Association for Computational Linguistics. find
  13. Micha Elsner and Eugene Charniak. 2008. You talking to me? a corpus and algorithm for conversation disentanglement. In Proceedings of ACL-08: HLT, June. find
  14. Michel Galley, Eric Fosler-Lussier, and Alexandros Potamianos. 2001. Hybrid natural language generation for spoken dialogue systems. In Proceedings of the 7th European Conference on Speech Communication and Technology (EUROSPEECH–01), pages 1735–1738, Aalborg, Denmark, September. find
  15. Jon Hasselgren, Erik Montnemery, Pierre Nugues, and Markus Svensson. 2003. Hms: a predictive text entry method using bigrams. In Proceedings of the 2003 EACL Workshop on Language Modeling for Text Entry Methods, TextEntry ’03. find
  16. Jerry R. Hobbs. 1985. On the coherence and structure of discourse. find
  17. Charles Lee Isbell, Jr., Michael J. Kearns, Dave Kormann, Satinder P. Singh, and Peter Stone. 2000. Cobot in lambdamoo: A social statistics agent. In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intellig… find
  18. Sina Jafarpour and Christopher J. C. Burges. 2010. Filter, rank, and transfer the knowledge: Learning to chat. find
  19. Howard Johnson, Joel Martin, George Foster, and Roland Kuhn. 2007. Improving translation quality by discarding most of the phrasetable. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 967… find our card
  20. Sangkeun Jung, Cheongjae Lee, Kyungduk Kim, Minwoo Jeong, and Gary Geunbae Lee. 2009. Data-driven user simulation for automated evaluation of spoken dialog systems. Comput. Speech Lang., 23:479–509, October. find
  21. Kevin Knight and Vasileios Hatzivassiloglou. 1995. Two-level, many-paths generation. In Proceedings of the 33rd annual meeting on Association for Computational Linguistics, ACL ’95. find
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL.… find
  23. J R Landis and G G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics. find
  24. Brian Langner, Stephan Vogel, and Alan W. Black. 2010. Evaluating a dialog language generation system: comparing the mountain system to other nlg approaches. In INTERSPEECH. find
  25. Anton Leuski and David R. Traum. 2010. Practical language processing for virtual humans. In Twenty-Second Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-10). find
  26. Bill MacCartney, Michel Galley, and Christopher D. Manning. 2008. A phrase-based alignment model for natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 802–811, Morristown, NJ, USA. Association for Computational Linguist… find
  27. Robert C. Moore. 2004. On log-likelihood-ratios and the significance of rare events. In EMNLP. find
  28. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  29. F. J. Och. 2003. Minimum error rate training for statistical machine translation. In ACL, pages 160–167. find
  30. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL, pages 311–318. find our card
  31. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 142–149. find our card
  32. Owen Rambow, Srinivas Bangalore, and Marilyn Walker. 2001. Natural language generation in dialog systems. In Proceedings of the first international conference on Human language technology research, HLT ’01, pages 1–4, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  33. Adwait Ratnaparkhi. 2000. Trainable methods for surface natural language generation. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference. find
  34. Sujith Ravi, Andrei Broder, Evgeniy Gabrilovich, Vanja Josifovski, Sandeep Pandey, and Bo Pang. 2010. Automatic generation of bid phrases for online advertising. In Proceedings of the third ACM international conference on Web search and data mining, WSDM ’10. find
  35. Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal, and Yi Liu. 2007. Statistical machine translation for query expansion in answer retrieval. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 464–471, Prague, Czech Republic, June… find
  36. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 172–180, Morristown, NJ, USA. Association for Com… find our card
  37. Samira Shaikh, Tomek Strzalkowski, Sarah Taylor, and Nick Webb. 2010. Vca: an experiment with a multiparty virtual chat agent. In Proceedings of the 2010 Workshop on Companionable Dialogue Systems. find
  38. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  39. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL on Main conference poster sessions, COLING-ACL ’06. find
  40. Xu Sun, Jianfeng Gao, Daniel Micol, and Chris Quirk. 2010. Learning phrase-based spelling error models from clickthrough data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 266–274, Morristown, NJ, USA. Association for Computational Lingui… find
  41. Reid Swanson and Andrew S. Gordon. 2008. Say anything: A massively collaborative open domain story writing companion. In Proceedings of the 1st Joint International Conference on Interactive Digital Storytelling: Interactive Storytelling, ICIDS ’08, pages 32– 40, Berlin, Heidelberg. Springer-Verlag. find
  42. Lidan Wang and Douglas W. Oard. 2009. Context-based message expansion for disentanglement of interleaved text conversations. In HLT-NAACL. find
  43. Joseph Weizenbaum. 1966. Eliza: a computer program for the study of natural language communication between man and machine. Commun. ACM, 9:36–45, January. find
  44. Yorick Wilks. 2006. Artificial companions as a new kind of interface to the future internet. In OII Research Report No. 13. find
  45. Yuk Wah Wong and Raymond Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference. find
  46. Yuk Wah Wong and Raymond Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/