Elamur Research⌕
☰
home / papers / 2023 / Factuality Challenges in the Era of Large Language Models

Factuality Challenges in the Era of Large Language Models

Summary

This perspective article, written by researchers and fact-checking practitioners, looks at how chatbots built on LLMs such as ChatGPT, Bing Chat and Bard affect the reliability of public information. It presents no new dataset or model. Instead, it surveys existing evidence and frames the problem in two parts:

  • Unintended errors: well-meant systems can hallucinate or produce poorly sourced answers, delivered in a confident tone, on sensitive topics like health and finance.
  • Deliberate misuse: malicious actors can use LLMs for scams, fake news sites and bot networks at scale.

The authors also note a feedback loop: disinformation produced with GenAI can end up in future training data. They consider where LLMs might help fact-checkers, such as summarization, stance detection and chat over curated corpora, and point out the limits of each. They close with an agenda organized around four areas: international coordination, regulation, AI literacy and technical safeguards. The paper matters because fact-checkers, newsrooms and policymakers can use it as a structured map of risks and countermeasures.

Key points

  • Evidence on undersourcing: in a cited study of generative search engines, close to 50% of generated sentences had no citation. Of the sentences that did cite something, only about 75% were actually backed by their source.
  • An analysis of 6,594 real chatbot conversations during the pandemic found that 30% of users searched with the keyword "COVID-19". The authors see this as risky because chatbots cope poorly with emerging or contested topics.
  • Fact-checks written by ChatGPT, even accurate ones, did not help people judge headline accuracy, and in some cases they strengthened belief in false claims.
  • Real incidents are cited, including Bard's incorrect claim about a James Webb telescope exoplanet image and Replika spreading a Bill Gates COVID-19 conspiracy theory.
  • The recommended safeguards are:
    • guardrails in chatbot conversations;
    • evidence-backed answers;
    • retrieval-augmented LLMs and knowledge graphs for factual consistency;
    • algorithms that detect coordinated campaigns;
    • AI literacy programs for all age groups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

70 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s42256-024-00881-z · arXiv 2310.05189

@misc{augenstein2023factuality,
  title = {{Factuality Challenges in the Era of Large Language Models}},
  author = {Augenstein, Isabelle and Baldwin, Timothy and Cha, Meeyoung and Chakraborty, Tanmoy and Ciampaglia, Giovanni Luca and Corney, David and DiResta, Renee and Ferrara, Emilio and Hale, Scott and Halevy, Alon and Hovy, Eduard and Ji, Heng and Menczer, Filippo and Miguez, Ruben and Nakov, Preslav and Scheufele, Dietram and Sharma, Shivam and Zagni, Giovanni},
  year = {2023},
  doi = {10.1038/s42256-024-00881-z},
  eprint = {2310.05189},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s42256-024-00881-z}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shannon, C. E. 10.1002/j.1538-7305.1948.tb01338.x (1948). The Bell Syst. Tech. J. 27, 379–423, DOI: — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) find
  2. Huang, J. & Chang, K. C.-C. Towards Reasoning in Large Language Models: A Survey. In Findings of the Association for Computational Linguistics: ACL 2023, 1049–1065, DOI: 10.18653/v1/2023.findings-acl.67 (Association for Computational Linguistics, Toronto, Canada, 2023). find our card
  3. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. et al. Improving language understanding by generative pre-training (2018). \#\# 4. OpenAI. GPT-4 technical report. arXiv:2303.08774 (2023). find our card
  4. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). find
  5. Introducing ChatGPT — openai.com. https://openai.com/blog/chatgpt. [Accessed 27-09-2023]. ↗
  6. Hu, K. ChatGPT sets record for fastest-growing user base - analyst note. Reuters (2023). find
  7. Thoppilan, R. et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022). find our card
  8. Taori, R. et al. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca (2023). ↗
  9. Vicuna: An open-source chatbot impressing GPT-4 https://lmsys.org/blog/2023-03-30-vicuna/. [Accessed 15-09-2023]. with 90%* ChatGPT quality. ↗
  10. Introducing Claude. https://www.anthropic.com/index/introducing-claude. [Accessed 15-09-2023]. \#\# 12. Falcon. https://falconllm.tii.ae/falcon.html. [Accessed 15-09-2023]. ↗
  11. Jurassic-2 models. https://docs.ai21.com/docs/jurassic-2-models. [Accessed 20-09-2023]. ↗
  12. Sengupta, N. et al. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint 2308.16149 (2023). 2308.16149. find our card
  13. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019). find
  14. Brown, T. et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, vol. 33, 1877–1901 (Curran Associates, Inc., 2020). find
  15. Zhao, W. X. et al. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). find our card
  16. Li, S. et al. Open-domain hierarchical event schema induction by incremental prompting and verification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5677–5697, DOI: 10.18653/v1/2023.acl-long.312 (Association for Computational Li… find
  17. Wei, J. et al. Emergent abilities of large language models. Trans. Mach. Learn. Res. 2022 (2022). find our card
  18. Bang, Y. et al. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. Preprint 2302.04023, arXiv (2023). DOI: 10.48550/arXiv.2302.04023. find our card
  19. Bergstrom, C. T. & Ogbunu, C. B. ChatGPT isn’t ‘hallucinating.’ It’s Bullshitting. https://undark.org/2023/04/06/chatgpt-isnt-hallucinating-its-bullshitting/ (2023). Accessed 2 October 2023. ↗
  20. Sison, A. J. G., Daza, M. T., Gozalo-Brizuela, R. & Garrido-Merchán, E. C. ChatGPT: More Than a “Weapon of Mass Deception” – Ethical Challenges and Responses from the Human-Centered Artificial Intelligence (HCAI) Perspective. Int. J. Human-Computer Interact. 0, 1–20, DOI: 10.1080/10447318.2023.22259… — [doi:10.1080/10447318.2023.2225931](https://doi.org/10.1080/10447318.2023.2225931) find
  21. Iftikhar, L. et al. Docgpt: Impact of chatgpt-3 on health services as a virtual doctor. EC Paediatr. 12, 45–55 (2023). find
  22. Chin, H. et al. User-chatbot conversations during the COVID-19 pandemic: Study based on topic modeling and sentiment analysis. J. Med. Internet Res. 25, e40922, DOI: 10.2196/40922 (2023). — [doi:10.2196/40922](https://doi.org/10.2196/40922) find
  23. Palanica, A., Flaschner, P., Thommandram, A., Li, M. & Fossat, Y. Physicians’ perceptions of chatbots in health care: Cross-sectional web-based survey. J Med Internet Res 21, e12887, DOI: 10.2196/12887 (2019). — [doi:10.2196/12887](https://doi.org/10.2196/12887) find
  24. Peskoff, D. & Stewart, B. Credible without credit: Domain experts assess generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 427–438, DOI: 10.18653/v1/2023.acl-short.37 (Association for Computational Linguis… find
  25. Srivastava, B. Did chatbots miss their “Apollo moment”? Potential, gaps, and lessons from using collaboration assistants during COVID-19. Patterns 2, 100308, DOI: https://doi.org/10.1016/j.patter.2021.100308 (2021). — [doi:10.1016/j.patter.2021.100308](https://doi.org/10.1016/j.patter.2021.100308) ↗
  26. Menczer, F., Crandall, D., Ahn, Y.-Y. & Kapadia, A. Addressing the harms of AI-generated inauthentic content. Nat. Mach. Intell. 5, 678–680, DOI: 10.1038/s42256-023-00690-w (2023). — [doi:10.1038/s42256-023-00690-w](https://doi.org/10.1038/s42256-023-00690-w) find
  27. Patel, A. & Sattler, J. Creatively malicious prompt engineering. Tech. Rep., WithSecure Labs (2023). find
  28. Brewster, J., Wang, M. & Palmer, C. Plagiarism-Bot? How Low-Quality Websites Are Using AI to Deceptively Rewrite https://www.newsguardtech.com/misinformation-monitor/august-2023/ Content from Mainstream News Outlets. (2023). [Accessed 23 Sep 2023]. ↗
  29. Yang, K.-C. & Menczer, F. Anatomy of an AI-powered malicious social botnet. Preprint 2307.16336, arXiv (2023). DOI: 10.48550/arXiv.2307.16336. — [doi:10.48550/arxiv.2307.16336](https://doi.org/10.48550/arxiv.2307.16336) find
  30. Verma, P. & Oremus, W. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. Wash. Post (2023). find
  31. DeVerna, M. R., Yan, H. Y., Yang, K.-C. & Menczer, F. Artificial intelligence is ineffective and potentially harmful for fact checking. Preprint 2308.10800, arXiv (2023). DOI: 10.48550/arXiv.2308.10800. — [doi:10.48550/arxiv.2308.10800](https://doi.org/10.48550/arxiv.2308.10800) find
  32. Ferrara, E. The history of digital spam. Commun. ACM 62, 82–91, DOI: 10.1145/3299768 (2019). \#\# 35. Vincent, J. Google’s AI chatbot Bard makes factual error in first demo (2023). \#\# 36. Marcus, G. Deep Learning Is Hitting a Wall (2022). 37. Ferrara, E. Should ChatGPT be Biased? Challenges and Risks… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/augenstein-2023-factuality-challenges-era-large-language/