Elamur Research⌕
☰
home / papers / 2011 / Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Summary

This paper tackles deceptive opinion spam: fake reviews written to read like genuine customer experiences, as opposed to obvious junk such as ads or off-topic text. Research on this problem had been held back by the lack of reliable labelled data. The authors built the first large public gold-standard corpus of 800 positive reviews for 20 popular Chicago hotels:

  • 400 truthful 5-star reviews mined from TripAdvisor.
  • 400 fake reviews commissioned on Amazon Mechanical Turk.

They compared three ways of framing detection:

  • n-gram text categorization,
  • psycholinguistic cues counted with LIWC,
  • genre identification based on part-of-speech distributions.

Human judges performed at roughly chance level. Automatic classifiers did much better. n-gram models were the strongest single approach, and combining LIWC with bigram features in an SVM reached 89.8% cross-validated accuracy. The work matters for two reasons. It supplies a benchmark that earlier heuristic studies lacked, and its feature analysis suggests that deception cues depend on the liar's context and goal rather than forming a universal list.

Key points

  • Data collection:
    • Fake reviews came from 400 AMT HITs spread evenly across 20 hotels. Each Turker could submit once, had to be US-based with at least a 90% approval rating, had up to 30 minutes, and was paid $1.
    • Collection took about 14 days.
    • Truthful reviews excluded very short texts and reviews by first-time TripAdvisor authors, of which 1,607 were removed.
  • Human performance: Human judges were mostly at chance, which matches earlier deception research.
  • Model results:
    • LIWC+BIGRAMS+SVM reached 89.8% accuracy. This was not significantly better than BIGRAMS+ SVM alone.
    • Even plain UNIGRAMS outperformed every approach that was not based on text categorization.
  • What distinguished truthful reviews: They used more concrete, sensory and spatial language, such as bathroom, small and location.
  • What distinguished deceptive reviews: They focused on things outside the hotel, such as husband, business and vacation. They also used more positive emotion words and more first person singular, which runs against earlier psycholinguistic findings.

Interesting ideas

  • I1. Crowdsource the negative class through role-play. Asking workers to pretend they are hotel marketers produced gold-standard fake text. This approach could create labelled data for other hard-to-observe phenomena, such as fake news or synthetic complaints.
  • I2. Lower the risk of label noise through sampling. The authors took "truthful" reviews only from the most-reviewed hotels and excluded first-time authors, because such reviews are less likely to hide spam. This is a cheap trick wherever a "clean" class is scraped from the wild.
  • I3. Benchmark against humans. Measuring human judges, not just a random baseline, shows whether a task truly needs automation. It also warns against relying on human-judged quality labels.
  • I4. Compare theory-driven and data-driven features on the same data. Pitting lexicon-based features (LIWC) against n-grams and genre-based POS features on one dataset, then inspecting the learned weights, turns a classifier into a tool for testing hypotheses. Examples include spatial detail in honest accounts and context-specific deception cues.
  • I5. Treat genre as a lens. Comparing POS distributions with known genre profiles (imaginative vs. informative writing) is a lightweight way to characterise any new text class, and could transfer to other languages and corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

127 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 121 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ott2011finding,
  title = {{Finding Deceptive Opinion Spam by Any Stretch of the Imagination}},
  author = {Ott, Myle and Choi, Yejin and Cardie, Claire and Hancock, Jeffrey T.},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1032/}
}

References

44 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Akkaya, A. Conrad, J. Wiebe, and R. Mihalcea. 2010. Amazon mechanical turk for subjectivity word sense disambiguation. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazons Mechanical Turk, Los Angeles, pages 195–203. find
  2. D. Biber, S. Johansson, G. Leech, S. Conrad, E. Finegan, and R. Quirk. 1999. Longman grammar of spoken and written English, volume 2. MIT Press. find
  3. C.F. Bond and B.M. DePaulo. 2006. Accuracy of deception judgments. Personality and Social Psychology Review, 10(3):214. find
  4. D.B. Buller and J.K. Burgoon. 1996. Interpersonal deception theory. Communication Theory, 6(3):203– 242. find
  5. W.L. Cade, B.A. Lehman, and A. Olney. 2010. An exploration of off topic conversation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 669–672. Association for Computational Linguistics. find
  6. S.F. Chen and J. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of the 34th annual meeting on Association for Computational Linguistics, pages 310–318. Association for Computational Linguistics. find our card
  7. C. Danescu-Niculescu-Mizil, G. Kossinets, J. Kleinberg, and L. Lee. 2009. How opinions are received by online communities: a case study on amazon.com helpfulness votes. In Proceedings of the 18th international conference on World wide web, pages 141–150. ACM. find
  8. H. Drucker, D. Wu, and V.N. Vapnik. 2002. Support vector machines for spam categorization. Neural Networks, IEEE Transactions on, 10(5):1048–1054. find
  9. G. Forman and M. Scholz. 2009. Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement. ACM SIGKDD Explorations, 12(1):49–57. find
  10. Z. Gyöngyi, H. Garcia-Molina, and J. Pedersen. 2004. Combating web spam with trustrank. In Proceedings of the Thirtieth international conference on Very large data bases-Volume 30, pages 576–587. VLDB Endowment. find
  11. J.T. Hancock, L.E. Curry, S. Goorha, and M. Woodworth. 2008. On lying and being lied to: A linguistic analysis of deception in computer-mediated communication. Discourse Processes, 45(1):1–23. find
  12. J. Jansen. 2010. Online product research. Pew Internet & American Life Project Report. find
  13. N. Jindal and B. Liu. 2008. Opinion spam and analysis. In Proceedings of the international conference on Web search and web data mining, pages 219–230. ACM. find
  14. T. Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. Machine Learning: ECML-98, pages 137–142. find our card
  15. T. Joachims. 1999. Making large-scale support vector machine learning practical. In Advances in kernel methods, page 184. MIT Press. find
  16. M.K. Johnson and C.L. Raye. 1981. Reality monitoring. Psychological Review, 88(1):67–85. find
  17. S.M. Kim, P. Pantel, T. Chklovski, and M. Pennacchiotti. 2006. Automatically assessing review helpfulness. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 423– 430. Association for Computational Linguistics. find
  18. D. Klein and C.D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 423–430. Association for Computational Linguistics. find our card
  19. J.R. Landis and G.G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159. find
  20. E.P. Lim, V.A. Nguyen, N. Jindal, B. Liu, and H.W. Lauw. 2010. Detecting product review spammers using rating behaviors. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 939–948. ACM. find
  21. S.W. Litvin, R.E. Goldsmith, and B. Pan. 2008. Electronic word-of-mouth in hospitality and tourism management. Tourism management, 29(3):458–468. find
  22. F. Mairesse, M.A. Walker, M.R. Mehl, and R.K. Moore. 2007. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of Artificial Intelligence Research, 30(1):457–500. find
  23. R. Mihalcea and C. Strapparava. 2009. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 309–312. Association for Computational Linguistics. find
  24. M.L. Newman, J.W. Pennebaker, D.S. Berry, and J.M. Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and Social Psychology Bulletin, 29(5):665. find
  25. A. Ntoulas, M. Najork, M. Manasse, and D. Fetterly. 2006. Detecting spam web pages through content analysis. In Proceedings of the 15th international conference on World Wide Web, pages 83–92. ACM. find
  26. M.P. O’Mahony and B. Smyth. 2009. Learning to recommend helpful hotel reviews. In Proceedings of the third ACM conference on Recommender systems, pages 305–308. ACM. find
  27. F. Peng and D. Schuurmans. 2003. Combining naive Bayes and n-gram language models for text classification. Advances in Information Retrieval, pages 547– 547. find
  28. J.W. Pennebaker, C.K. Chung, M. Ireland, A. Gonzales, and R.J. Booth. 2007. The development and psychometric properties of LIWC2007. Austin, TX, LIWC. Net. find
  29. N. Quadrianto, A.J. Smola, T.S. Caetano, and Q.V. Le. 2009. Estimating labels from label proportions. The Journal of Machine Learning Research, 10:2349– 2374. find
  30. P. Rayson, A. Wilson, and G. Leech. 2001. Grammatical word class variation within the British National Corpus sampler. Language and Computers, 36(1):295– 306. find
  31. R.A. Rigby and D.M. Stasinopoulos. 2005. Generalized additive models for location, scale and shape. Journal of the Royal Statistical Society: Series C (Applied Statistics), 54(3):507–554. find
  32. F. Sebastiani. 2002. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1–47. find
  33. M.Á. Serrano, A. Flammini, and F. Menczer. 2009. Modeling statistical properties of written text. PloS one, 4(4):5372. find
  34. A. Stolcke. 2002. SRILM-an extensible language modeling toolkit. In Seventh International Conference on Spoken Language Processing, volume 3, pages 901– 904. Citeseer. find
  35. C. Toma and J.T. Hancock. In Press. What Lies Beneath: The Linguistic Traces of Deception in Online Dating Profiles. Journal of Communication. find
  36. A. Vrij, S. Mann, S. Kristen, and R.P. Fisher. 2007. Cues to deception and ability to detect lies as a function of police interview styles. Law and human behavior, 31(5):499–518. find
  37. A. Vrij, S. Leal, P.A. Granhag, S. Mann, R.P. Fisher, J. Hillman, and K. Sperry. 2009. Outsmarting the liars: The benefit of asking unanticipated questions. Law and human behavior, 33(2):159–166. find
  38. A. Vrij. 2008. Detecting lies and deceit: Pitfalls and opportunities. Wiley-Interscience. find
  39. W. Weerkamp and M. De Rijke. 2008. Credibility improves topical blog post retrieval. ACL-08: HLT, pages 923–931. find
  40. M. Weimer, I. Gurevych, and M. Mühlhäuser. 2007. Automatically assessing the post quality in online discussions on software. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 125–128. Association for Computational Linguistics. find
  41. G. Wu, D. Greene, B. Smyth, and P. Cunningham. 2010. Distortion as a validation criterion in the identification of suspicious reviews. Technical report, UCD-CSI-2010-04, University College Dublin. find
  42. K.H. Yoo and U. Gretzel. 2009. Comparison of Deceptive and Truthful Travel Reviews. Information and Communication Technologies in Tourism 2009, pages 37–47. find
  43. L. Zhou, J.K. Burgoon, D.P. Twitchell, T. Qin, and J.F. Nunamaker Jr. 2004. A comparison of classification methods for predicting deception in computer-mediated communication. Journal of Management Information Systems, 20(4):139–166. find
  44. L. Zhou, Y. Shi, and D. Zhang. 2008. A Statistical Language Modeling Approach to Online Deception Detection. IEEE Transactions on Knowledge and Data Engineering, 20(8):1077–1081. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ott-2011-finding-deceptive-opinion-spam-any/