Elamur Research⌕
☰
home / papers / 2023 / On the Risk of Misinformation Pollution with Large Language Models

On the Risk of Misinformation Pollution with Large Language Models

Summary

This Findings of EMNLP 2023 paper asks how easily modern LLMs can be turned into producers of plausible false content, and what happens when such text enters the corpora that retrieval-based Open-Domain Question Answering (ODQA) systems use. The authors build a threat model that covers two cases: accidental pollution from hallucinations and deliberate disinformation written to push a chosen false answer. They use GPT-3.5 (text-davinci-003) zero-shot under four prompting regimes, inject the generated passages into the retrieval corpus, and measure how the QA systems respond. The experiments mix two retrievers with two test datasets. Synthetic misinformation sharply lowers ODQA accuracy, by up to 87%. Text that is persuasive to humans does not always fool machines, and the reverse also holds. The authors test three countermeasures: misinformation detection, vigilant prompting, and a reader ensemble based on majority voting. Each helps to some degree but has a cost. The work matters because it shows that cheap, automated corpus poisoning is a realistic risk for knowledge-intensive NLP systems.

Key points

  • The four generation settings are GenRead, CtrlGen, Revise and Reit.
    • GenRead asks the model for a background document, so any falsehoods come from hallucination.
    • CtrlGen writes a supporting article for a predefined false claim.
    • Revise makes minimal edits to a real article so that it supports the false claim.
    • Reit produces text aimed at manipulating QA models rather than at looking realistic to readers.
  • Polluted corpora cut ODQA performance by as much as 87%, measured as relative EM change.
  • One fake document of about 200 words costs an estimated $0.01–$0.04 through the text-davinci-003 API, far less than paying human writers.
  • When GPT is used as the reader, its context is capped at 10 passages because of input-length limits.
  • The prompts and generated documents are publicly released at github.com/MexicanLemonade/LLM-Misinfo-QA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.97

@inproceedings{pan2023risk,
  title = {{On the Risk of Misinformation Pollution with Large Language Models}},
  author = {Pan, Yikang and Pan, Liangming and Chen, Wenhu and Nakov, Preslav and Kan, Min-Yen and Wang, William},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.97},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.97}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aaditya Bhat. 2023. Gpt-wiki-intro (revision 0e458f5). find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Ben Buchanan, Andrew Lohn, and Micah Musser. 2021. Truth, lies, and automation: How language models could change disinformation. Center for Security and Emerging Technology. find
  4. Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum 7 https://thegradient.pub/ why-we-released-grover/ Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. 2023. Poisoning Web-Scale Training Datasets is Practical. ArXiv:2302.10149 [cs]. — [arXiv:2302.10149](https://arxiv.org/abs/2302.10149) ↗
  5. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023. On the Possibilities of AI-Generated Text Detection. ArXiv:2304.04736 [cs]. — [arXiv:2304.04736](https://arxiv.org/abs/2304.04736) find
  6. Hung-Ting Chen, Michael J. Q. Zhang, and Eunsol Choi. 2022. Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence. ArXiv:2210.13701 [cs]. find
  7. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021a. All that’s ’human’ is not gold: Evaluating human evaluation of generated text. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7282–7296. find
  8. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021b. All That’s ’Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text. ArXiv:2107.00061 [cs]. find
  9. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. ArXiv:2109.10086 [cs]. find
  10. Yi R. Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen R. McKeown, Mohit Bansal, and Avi Sil. 2021. Infosurgeon: Cross-media fine-grained information consistency checking for fake news detection. In Annual Meeting of the Association for Computational… find
  11. Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. CoRR, abs/2301.04246. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  12. Shane Greenstein and Feng Zhu. 2012. Is Wikipedia Biased? American Economic Review, 102(3):343– 348. find
  13. Maurício Gruppi, Benjamin D. Horne, and Sibel Adalı. 2022. NELA-GT-2021: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles. ArXiv:2203.05659 [cs]. — [arXiv:2203.05659](https://arxiv.org/abs/2203.05659) find
  14. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning, pages 3929–3938. PMLR. ISSN: 2640-3498. find
  15. Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2022. FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation. ArXiv:2209.14290 [cs]. — [arXiv:2209.14290](https://arxiv.org/abs/2209.14290) find
  16. Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Jiyoung Whang. 2023. Discern and Answer: Mitigating the Impact of Misinformation in Retrieval-Augmented Models with Discriminators. ArXiv:2305.01579 [cs]. — [arXiv:2305.01579](https://arxiv.org/abs/2305.01579) find
  17. Kung-Hsiang Huang, Kathleen R. McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2022. Faking fake news for real fake news detection: Propaganda-loaded training data generation. CoRR, abs/2203.05386. find
  18. Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. ArXiv:2007.01282 [cs]. find our card
  19. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, and Pascale Fung. 2022. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, page 3571730. ArXiv:2202.03629 [cs]. find our card
  20. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl… find our card
  21. Sarah Kreps, R Miles McCain, and Miles Brundage. 2022. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation. Journal of experimental political science, 9(1):104– 117. find
  22. Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes. In Proceedings of the 25th International Conference on World Wide Web, pages 591–602, Montréal Québec Canada. International World Wide Web Conferences Steering C… find
  23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  24. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. ArXiv:2005.11401 [cs]. find
  25. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023a. Multi-step Jailbreaking Privacy Attacks on ChatGPT. ArXiv:2304.05197 [cs]. find our card
  26. Xinyi Li, Yongfeng Zhang, and Edward C. Malthouse. 2023b. A Preliminary Study of ChatGPT on News Recommendation: Personalization, Provider Fairness, Fake News. ArXiv:2306.10702 [cs]. — [arXiv:2306.10702](https://arxiv.org/abs/2306.10702) find
  27. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Develop… find
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv:1907.11692 [cs]. find
  29. Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A Benchmark for Adaptation to New Knowle… — [arXiv:2205.11388](https://arxiv.org/abs/2205.11388) find
  30. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2022. Entity-Based Knowledge Conflicts in Question Answering. ArXiv:2109.05052 [cs]. find our card
  31. Alexandra Luccioni and Joseph Viviano. 2021. What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  32. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. ArXiv:2301.11305 [cs]. find our card
  33. OpenAI. 2023. GPT-4 technical report. abs/2303.08774. find our card
  34. CoRR, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 20… find
  35. Liangming Pan, Wenhu Chen, Min-Yen Kan, and William Yang Wang. 2021. ContraQA: Question Answering under Contradicting Contexts. ArXiv:2110.07803 [cs]. — [arXiv:2110.07803](https://arxiv.org/abs/2110.07803) find
  36. Emmanouil Papadogiannakis, Panagiotis Papadopoulos, Evangelos P. Markatos, and Nicolas Kourtellis. 2023. Who Funds Misinformation? A Systematic Analysis of the Ad-related Profit Routines of Fake News sites. ArXiv:2202.05079 [cs]. — [arXiv:2202.05079](https://arxiv.org/abs/2202.05079) find
  37. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv:1910.10683 [cs, stat]. find
  38. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  39. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-Generated Text be Reliably Detected? ArXiv:2303.11156 [cs]. find our card
  40. Zhihong Shao and Minlie Huang. 2022. Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework. ArXiv:2110.08544 [cs]. find
  41. Harald Stiff and Fredrik Johansson. 2022. Detecting computer-generated disinformation. International Journal of Data Science and Analytics, 13(4):363– 383. find
  42. Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2022. Context Generation Improves Open Domain Question Answering. ArXiv:2210.06349 [cs]. find
  43. Orion Weller, Aleem Khan, Nathaniel Weir, Dawn Lawrie, and Benjamin Van Durme. 2022. Defending Against Poisoning Attacks in Open-Domain Question Answering. ArXiv:2212.10002 [cs]. — [arXiv:2212.10002](https://arxiv.org/abs/2212.10002) find
  44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find our card
  45. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than Retrieve: Large Language Models are Strong Context Generators. ArXiv:2209.10063 [cs]. find our card
  46. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 9051–9062. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2023-risk-misinformation-pollution-large-language/